[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:18 dist-test-slave-t5lt krb5kdc[22328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:18 dist-test-slave-t5lt krb5kdc[22328](info): set up 2 sockets
Sep 23 15:11:18 dist-test-slave-t5lt krb5kdc[22328](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 15:11:19.463826 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:19 dist-test-slave-t5lt krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:19.602180 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40581
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:40581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:19.685364 22341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:19.685652 22341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:19.685696 22341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:19.687054 22341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:19.687139 22341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:19.687167 22341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:19.687201 22341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:19.687234 22341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:19.688865 22341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:40581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:19.689188 22341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:19.689494 22341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:19.696409 22347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:19.697108 22346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:19.697194 22349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:19.697492 22341 server_base.cc:1034] running on GCE node
I20260923 15:11:19.697836 22341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:19.698292 22341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:19.699509 22341 hybrid_clock.cc:648] HybridClock initialized: now 1790176279699509 us; error 58 us; skew 500 ppm
Sep 23 15:11:19 dist-test-slave-t5lt krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176279, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:19.705778 22341 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:19.706311 22341 webserver.cc:460] Webserver started at http://127.21.204.126:34741/ using document root <none> and password file <none>
I20260923 15:11:19.706634 22341 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:19.706707 22341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:19.706872 22341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:19.707986 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "d45516bfb90b4f9b89bda98c4c9f181d"
format_stamp: "Formatted at 2026-09-23 15:11:19 on dist-test-slave-t5lt"
server_key: "73b5277a7b543f1b58bdd83b6cde2f8c"
server_key_iv: "2b5faf9246bb545c6c51fea0f98764a5"
server_key_version: "encryptionkey@0"
I20260923 15:11:19.708366 22341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "d45516bfb90b4f9b89bda98c4c9f181d"
format_stamp: "Formatted at 2026-09-23 15:11:19 on dist-test-slave-t5lt"
server_key: "73b5277a7b543f1b58bdd83b6cde2f8c"
server_key_iv: "2b5faf9246bb545c6c51fea0f98764a5"
server_key_version: "encryptionkey@0"
I20260923 15:11:19.709849 22341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260923 15:11:19.710816 22356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:19.711138 22341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:19.711299 22341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "d45516bfb90b4f9b89bda98c4c9f181d"
format_stamp: "Formatted at 2026-09-23 15:11:19 on dist-test-slave-t5lt"
server_key: "73b5277a7b543f1b58bdd83b6cde2f8c"
server_key_iv: "2b5faf9246bb545c6c51fea0f98764a5"
server_key_version: "encryptionkey@0"
I20260923 15:11:19.711458 22341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:19.725245 22341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:19.728101 22341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:19.728286 22341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:19.732602 22341 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:40581
I20260923 15:11:19.732650 22408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:40581 every 8 connection(s)
I20260923 15:11:19.733036 22341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:19.734069 22409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:19.736421 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d: Bootstrap starting.
I20260923 15:11:19.737010 22409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:19.737349 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 22341
I20260923 15:11:19.737438 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:19.737535 22409 log.cc:824] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:19.738134 22321 external_mini_cluster.cc:1428] Setting key 599f0d50517e15317297f21146f405a6
I20260923 15:11:19.738422 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d: No bootstrap required, opened a new log
I20260923 15:11:19.740300 22409 raft_consensus.cc:348] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40581 } }
I20260923 15:11:19.740469 22409 raft_consensus.cc:374] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:19.740517 22409 raft_consensus.cc:729] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45516bfb90b4f9b89bda98c4c9f181d, State: Initialized, Role: FOLLOWER
I20260923 15:11:19.740665 22409 consensus_queue.cc:260] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40581 } }
I20260923 15:11:19.740751 22409 raft_consensus.cc:388] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:19.740794 22409 raft_consensus.cc:482] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:19.740844 22409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:19.742156 22409 raft_consensus.cc:504] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40581 } }
I20260923 15:11:19.742302 22409 leader_election.cc:302] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d45516bfb90b4f9b89bda98c4c9f181d; no voters:
I20260923 15:11:19.742514 22409 leader_election.cc:288] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:19.742535 22414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:19.742650 22414 raft_consensus.cc:686] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [term 1 LEADER]: Becoming Leader. State: Replica: d45516bfb90b4f9b89bda98c4c9f181d, State: Running, Role: LEADER
I20260923 15:11:19.742794 22414 consensus_queue.cc:237] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40581 } }
I20260923 15:11:19.742831 22409 sys_catalog.cc:564] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:19.743265 22416 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40581 } } }
I20260923 15:11:19.743356 22416 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:19.744203 22420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:19.744483 22415 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45516bfb90b4f9b89bda98c4c9f181d. Latest consensus state: current_term: 1 leader_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45516bfb90b4f9b89bda98c4c9f181d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40581 } } }
I20260923 15:11:19.744812 22415 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:19.744925 22420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:19.746682 22420 catalog_manager.cc:1347] Generated new cluster ID: 6956466a26d84af08d46a743e89e4a3d
I20260923 15:11:19.746735 22420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:11:19 dist-test-slave-t5lt krb5kdc[22328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:19.754567 22420 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:19.755155 22420 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:19.758206 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:19.739042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.21.204.126:40581)
0923 15:11:19.739230 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:19.739240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:19.739464 (+ 224us) server_negotiation.cc:407] Connection header received
0923 15:11:19.741353 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:19.741360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:19.741425 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:19.741614 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:19.743992 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:19.745680 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:19.747394 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:19.748078 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:19.748221 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:19.753515 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:19.753520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:19.753521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:19.753535 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:19.755926 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:19.756472 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:19.756475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:19.756475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:19.756622 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:19.756952 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:19.756953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:19.756954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:19.757272 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:19.757446 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:19.757752 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 15:11:19.757935 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260923 15:11:19.768926 22420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d45516bfb90b4f9b89bda98c4c9f181d: Generated new TSK 0
I20260923 15:11:19.769160 22420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:19.804857 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40581
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:19.890789 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:19.891104 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:19.891137 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:19.892343 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:19.892387 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:19.892414 22435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:19.893951 22435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:19.894203 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:19.894451 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:19.900228 22443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:19.900270 22435 server_base.cc:1034] running on GCE node
W20260923 15:11:19.900252 22441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:19.900252 22440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:19.900610 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:19.900873 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:19.902043 22435 hybrid_clock.cc:648] HybridClock initialized: now 1790176279902042 us; error 37 us; skew 500 ppm
Sep 23 15:11:19 dist-test-slave-t5lt krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176279, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:19.904650 22435 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:19.905170 22435 webserver.cc:460] Webserver started at http://127.21.204.65:44799/ using document root <none> and password file <none>
I20260923 15:11:19.905408 22435 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:19.905476 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:19.905612 22435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:19.906726 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "b8dfa49d7c8442afa805eb6cabf06c48"
format_stamp: "Formatted at 2026-09-23 15:11:19 on dist-test-slave-t5lt"
server_key: "a4a58c9497ccf149f6d9323ce7d98a4a"
server_key_iv: "c1f336c14a2d0ff87ed9932fcd846d05"
server_key_version: "encryptionkey@0"
I20260923 15:11:19.907088 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "b8dfa49d7c8442afa805eb6cabf06c48"
format_stamp: "Formatted at 2026-09-23 15:11:19 on dist-test-slave-t5lt"
server_key: "a4a58c9497ccf149f6d9323ce7d98a4a"
server_key_iv: "c1f336c14a2d0ff87ed9932fcd846d05"
server_key_version: "encryptionkey@0"
I20260923 15:11:19.908504 22435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 15:11:19.909327 22450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:19.909524 22435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:19.909629 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "b8dfa49d7c8442afa805eb6cabf06c48"
format_stamp: "Formatted at 2026-09-23 15:11:19 on dist-test-slave-t5lt"
server_key: "a4a58c9497ccf149f6d9323ce7d98a4a"
server_key_iv: "c1f336c14a2d0ff87ed9932fcd846d05"
server_key_version: "encryptionkey@0"
I20260923 15:11:19.909713 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:19.930481 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:19.932547 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:19.932694 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:19.932950 22435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:19.933302 22435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:19.933359 22435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:19.933403 22435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:19.933431 22435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:19.939775 22435 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:45397
I20260923 15:11:19.939817 22563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:45397 every 8 connection(s)
I20260923 15:11:19.940157 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:19.940759 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 22435
I20260923 15:11:19.940863 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:19.941049 22321 external_mini_cluster.cc:1428] Setting key 8e8fa6bebde6db63dcf31816cdf3a060
Sep 23 15:11:19 dist-test-slave-t5lt krb5kdc[22328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176279, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:11:19.952044 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:19.940944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:43081 (local address 127.21.204.126:40581)
0923 15:11:19.941006 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:19.941008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:19.941287 (+ 279us) server_negotiation.cc:407] Connection header received
0923 15:11:19.941983 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:19.941984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:19.941997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:19.942060 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:19.943277 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:19.943800 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:19.945476 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:19.945740 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:19.945810 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:19.948533 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:19.948535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:19.948535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:19.948551 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:19.950208 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:19.950690 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:19.950692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:19.950692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:19.950769 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:19.951086 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:19.951087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:19.951088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:19.951214 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:19.951284 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:19.951853 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 15:11:19.951945 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 15:11:19.952127 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:19.940926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40581 (local address 127.21.204.65:43081)
0923 15:11:19.941118 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:19.941157 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:19.941859 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:19.942136 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:19.942140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:19.942855 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:19.943172 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:19.943180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:19.943894 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:19.943895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:19.945338 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:19.945342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:19.945840 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:19.945842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:19.946038 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:19.946622 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:19.946634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:19.948405 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:19.950307 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:19.950314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:19.950316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:19.950569 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:19.950868 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:19.950873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:19.950873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:19.950991 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:19.951307 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:19.951315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:19.951679 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 15:11:19.951805 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 15:11:19.951945 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
I20260923 15:11:19.952880 22564 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40581
I20260923 15:11:19.953002 22564 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:19.953217 22564 heartbeater.cc:511] Master 127.21.204.126:40581 requested a full tablet report, sending...
I20260923 15:11:19.953802 22373 ts_manager.cc:194] Registered new tserver with Master: b8dfa49d7c8442afa805eb6cabf06c48 (127.21.204.65:45397)
I20260923 15:11:19.954454 22373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:43081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:19.971061 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40581
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:20.064105 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:20.064380 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:20.064424 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:20.065763 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:20.065837 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:20.065887 22569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:20.067677 22569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:20.067979 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:20.068228 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:20.073503 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:20.073541 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:20.073691 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:20.073832 22569 server_base.cc:1034] running on GCE node
I20260923 15:11:20.074074 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:20.074325 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:20.075505 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790176280075496 us; error 33 us; skew 500 ppm
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:20.078329 22569 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:20.078748 22569 webserver.cc:460] Webserver started at http://127.21.204.66:35351/ using document root <none> and password file <none>
I20260923 15:11:20.078981 22569 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:20.079027 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:20.079190 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:20.080300 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "f1f4859fc55d47a68b2fef34e092c706"
format_stamp: "Formatted at 2026-09-23 15:11:20 on dist-test-slave-t5lt"
server_key: "152d131b94d2e8246d3fe6b2b1c7c29a"
server_key_iv: "9ac079dcdd91176183265c3d66947a4d"
server_key_version: "encryptionkey@0"
I20260923 15:11:20.080703 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "f1f4859fc55d47a68b2fef34e092c706"
format_stamp: "Formatted at 2026-09-23 15:11:20 on dist-test-slave-t5lt"
server_key: "152d131b94d2e8246d3fe6b2b1c7c29a"
server_key_iv: "9ac079dcdd91176183265c3d66947a4d"
server_key_version: "encryptionkey@0"
I20260923 15:11:20.082129 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:20.082904 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:20.083108 22569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:20.083201 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "f1f4859fc55d47a68b2fef34e092c706"
format_stamp: "Formatted at 2026-09-23 15:11:20 on dist-test-slave-t5lt"
server_key: "152d131b94d2e8246d3fe6b2b1c7c29a"
server_key_iv: "9ac079dcdd91176183265c3d66947a4d"
server_key_version: "encryptionkey@0"
I20260923 15:11:20.083259 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:20.096675 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:20.099124 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:20.099241 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:20.099428 22569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:20.099706 22569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:20.099740 22569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:20.099764 22569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:20.099779 22569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:20.108043 22569 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:46311
I20260923 15:11:20.108193 22697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:46311 every 8 connection(s)
I20260923 15:11:20.108486 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:20.117076 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 22569
I20260923 15:11:20.117197 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:20.117374 22321 external_mini_cluster.cc:1428] Setting key 3f073931bef8c20e4715cc989bede8b0
I20260923 15:11:20.119079 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:20.109377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:40053 (local address 127.21.204.126:40581)
0923 15:11:20.109501 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:20.109504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:20.109770 (+ 266us) server_negotiation.cc:407] Connection header received
0923 15:11:20.110680 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:20.110682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:20.110694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:20.110745 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:20.111820 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.112334 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.113317 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.113525 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.113615 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:20.116070 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:20.116071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:20.116071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:20.116077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:20.117550 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.118100 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.118101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.118102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.118166 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.118435 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.118435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.118436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.118550 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:20.118634 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:20.118905 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 15:11:20.118991 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 15:11:20.119318 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:20.109336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40581 (local address 127.21.204.66:40053)
0923 15:11:20.109620 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:20.109657 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:20.110502 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:20.110832 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:20.110835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:20.111516 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:20.111718 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:20.111724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.112433 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.112434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:20.113224 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:20.113227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.113628 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.113629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:20.113769 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:20.114365 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:20.114373 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:20.115975 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:20.117681 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:20.117689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:20.117691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:20.117996 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:20.118234 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:20.118235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:20.118236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:20.118358 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:20.118624 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:20.118631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:20.118771 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 15:11:20.118871 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 15:11:20.119028 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
I20260923 15:11:20.119855 22698 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40581
I20260923 15:11:20.119951 22698 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:20.120148 22698 heartbeater.cc:511] Master 127.21.204.126:40581 requested a full tablet report, sending...
I20260923 15:11:20.120594 22373 ts_manager.cc:194] Registered new tserver with Master: f1f4859fc55d47a68b2fef34e092c706 (127.21.204.66:46311)
I20260923 15:11:20.121174 22373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:40053
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:20.145567 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40581
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:20.247691 22703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:20.247895 22703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:20.247918 22703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:20.249081 22703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:20.249125 22703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:20.249181 22703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:20.250748 22703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40581
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:20.250993 22703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:20.251230 22703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:20.256664 22709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:20.256743 22711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:20.256685 22708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:20.257114 22703 server_base.cc:1034] running on GCE node
I20260923 15:11:20.257350 22703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:20.257550 22703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:20.258705 22703 hybrid_clock.cc:648] HybridClock initialized: now 1790176280258691 us; error 30 us; skew 500 ppm
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:20.261420 22703 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:20.262022 22703 webserver.cc:460] Webserver started at http://127.21.204.67:44599/ using document root <none> and password file <none>
I20260923 15:11:20.262280 22703 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:20.262351 22703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:20.262495 22703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:20.263744 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "ec9101e77c8440569c6183c402ca1692"
format_stamp: "Formatted at 2026-09-23 15:11:20 on dist-test-slave-t5lt"
server_key: "f41bf68d93668d428565edbb04b809f6"
server_key_iv: "e76588cc19aaedd0f7dc611d1c92deb6"
server_key_version: "encryptionkey@0"
I20260923 15:11:20.264169 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "ec9101e77c8440569c6183c402ca1692"
format_stamp: "Formatted at 2026-09-23 15:11:20 on dist-test-slave-t5lt"
server_key: "f41bf68d93668d428565edbb04b809f6"
server_key_iv: "e76588cc19aaedd0f7dc611d1c92deb6"
server_key_version: "encryptionkey@0"
I20260923 15:11:20.265684 22703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:20.266729 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:20.267007 22703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:20.267115 22703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "ec9101e77c8440569c6183c402ca1692"
format_stamp: "Formatted at 2026-09-23 15:11:20 on dist-test-slave-t5lt"
server_key: "f41bf68d93668d428565edbb04b809f6"
server_key_iv: "e76588cc19aaedd0f7dc611d1c92deb6"
server_key_version: "encryptionkey@0"
I20260923 15:11:20.267227 22703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:20.293272 22703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:20.295745 22703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:20.295928 22703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:20.296191 22703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:20.296511 22703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:20.296566 22703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:20.296634 22703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:20.296665 22703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:20.305320 22703 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:36151
I20260923 15:11:20.305408 22831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:36151 every 8 connection(s)
I20260923 15:11:20.305771 22703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:20.311163 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 22703
I20260923 15:11:20.311260 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:20.311445 22321 external_mini_cluster.cc:1428] Setting key de31dca7b94ca768af4fc7912e9223dc
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:20.316874 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:20.306677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:60435 (local address 127.21.204.126:40581)
0923 15:11:20.306782 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:20.306785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:20.307058 (+ 273us) server_negotiation.cc:407] Connection header received
0923 15:11:20.307614 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:20.307616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:20.307627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:20.307698 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:20.308683 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.309238 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.310484 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.310682 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.310792 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:20.313668 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:20.313670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:20.313670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:20.313677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:20.315344 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.315876 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.315877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.315878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.315938 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.316168 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.316168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.316169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.316279 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:20.316335 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:20.316732 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 15:11:20.316809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 15:11:20.316954 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:20.306616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40581 (local address 127.21.204.67:60435)
0923 15:11:20.306923 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:20.306961 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:20.307486 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:20.307775 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:20.307778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:20.308358 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:20.308567 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:20.308572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.309375 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.309377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:20.310359 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:20.310362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.310849 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.310850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:20.310992 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:20.311655 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:20.311665 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:20.313530 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:20.315521 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:20.315528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:20.315530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:20.315802 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:20.316001 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:20.316001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:20.316002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:20.316108 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:20.316361 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:20.316368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:20.316497 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 15:11:20.316644 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 15:11:20.316782 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":200,"threads_started":1}
I20260923 15:11:20.317842 22832 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40581
I20260923 15:11:20.318022 22832 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:20.318194 22832 heartbeater.cc:511] Master 127.21.204.126:40581 requested a full tablet report, sending...
I20260923 15:11:20.318619 22373 ts_manager.cc:194] Registered new tserver with Master: ec9101e77c8440569c6183c402ca1692 (127.21.204.67:36151)
I20260923 15:11:20.318966 22373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:60435
I20260923 15:11:20.324218 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:20.350659 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:20.341601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.21.204.65:45397)
0923 15:11:20.341926 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:20.341932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:20.341944 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:11:20.341979 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:20.341980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:20.341996 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:20.342120 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:20.342740 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.343917 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.344724 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.345156 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.345248 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:20.347739 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:20.347746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:20.347746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:20.347753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:20.349483 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.349819 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.349822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.349823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.349976 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.350170 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.350170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.350170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.350323 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:20.350457 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:20.350507 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 15:11:20.350577 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":95,"threads_started":1}
W20260923 15:11:20.351177 22476 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48574
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:20.373689 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:20.366438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.21.204.65:45397)
0923 15:11:20.366508 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:20.366509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:20.366567 (+ 58us) server_negotiation.cc:407] Connection header received
0923 15:11:20.366618 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:20.366619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:20.366635 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:20.366703 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:20.367231 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.367727 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.368509 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:20.368714 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:20.368770 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:20.371333 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:20.371334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:20.371335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:20.371342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:20.372746 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.373056 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.373056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.373057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.373105 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:20.373268 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:20.373269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:20.373269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:20.373354 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:20.373436 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:20.373507 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 15:11:20.373565 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 15:11:20.374229 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 22435
I20260923 15:11:20.379871 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 22569
I20260923 15:11:20.385332 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 22703
I20260923 15:11:20.389846 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 22341
2026-09-23T15:11:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1988 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22846](info): set up 2 sockets
Sep 23 15:11:20 dist-test-slave-t5lt krb5kdc[22846](info): commencing operation
krb5kdc: starting...
W20260923 15:11:21.451062 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:21 dist-test-slave-t5lt krb5kdc[22846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:21.577419 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34605
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:21.659540 22859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:21.659837 22859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:21.659876 22859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:21.661120 22859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:21.661193 22859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:21.661221 22859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:21.661252 22859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:21.661276 22859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:21.662881 22859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:21.663172 22859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:21.663403 22859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:21.668491 22867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:21.668596 22864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:21.668601 22865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:21.668780 22859 server_base.cc:1034] running on GCE node
I20260923 15:11:21.669008 22859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:21.669262 22859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:21.670399 22859 hybrid_clock.cc:648] HybridClock initialized: now 1790176281670382 us; error 42 us; skew 500 ppm
Sep 23 15:11:21 dist-test-slave-t5lt krb5kdc[22846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:21.672889 22859 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:21.673337 22859 webserver.cc:460] Webserver started at http://127.21.204.126:37359/ using document root <none> and password file <none>
I20260923 15:11:21.673564 22859 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:21.673617 22859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:21.673755 22859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:21.674808 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "81e66b0c616b45ee90bb471f4e62fb58"
format_stamp: "Formatted at 2026-09-23 15:11:21 on dist-test-slave-t5lt"
server_key: "665298d4b6448fadad4970160a1a0f19"
server_key_iv: "997422223c1bcac476e90cc156903b35"
server_key_version: "encryptionkey@0"
I20260923 15:11:21.675166 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "81e66b0c616b45ee90bb471f4e62fb58"
format_stamp: "Formatted at 2026-09-23 15:11:21 on dist-test-slave-t5lt"
server_key: "665298d4b6448fadad4970160a1a0f19"
server_key_iv: "997422223c1bcac476e90cc156903b35"
server_key_version: "encryptionkey@0"
I20260923 15:11:21.676492 22859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:11:21.677248 22874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:21.677462 22859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:21.677552 22859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "81e66b0c616b45ee90bb471f4e62fb58"
format_stamp: "Formatted at 2026-09-23 15:11:21 on dist-test-slave-t5lt"
server_key: "665298d4b6448fadad4970160a1a0f19"
server_key_iv: "997422223c1bcac476e90cc156903b35"
server_key_version: "encryptionkey@0"
I20260923 15:11:21.677652 22859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:21.689580 22859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:21.691524 22859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:21.691686 22859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:21.695832 22859 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34605
I20260923 15:11:21.695989 22926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34605 every 8 connection(s)
I20260923 15:11:21.696200 22859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:21.697058 22927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:21.699457 22927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58: Bootstrap starting.
I20260923 15:11:21.700237 22927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:21.700537 22927 log.cc:824] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:21.701297 22927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58: No bootstrap required, opened a new log
I20260923 15:11:21.702168 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 22859
I20260923 15:11:21.702242 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:21.702405 22321 external_mini_cluster.cc:1428] Setting key 4c78b2fe9c6ea58787635a3c20302533
I20260923 15:11:21.703399 22927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66b0c616b45ee90bb471f4e62fb58" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34605 } }
I20260923 15:11:21.703572 22927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:21.703612 22927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e66b0c616b45ee90bb471f4e62fb58, State: Initialized, Role: FOLLOWER
I20260923 15:11:21.703753 22927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66b0c616b45ee90bb471f4e62fb58" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34605 } }
I20260923 15:11:21.703841 22927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:21.703887 22927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:21.703940 22927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:21.704927 22927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66b0c616b45ee90bb471f4e62fb58" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34605 } }
I20260923 15:11:21.705020 22927 leader_election.cc:302] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81e66b0c616b45ee90bb471f4e62fb58; no voters:
I20260923 15:11:21.705204 22927 leader_election.cc:288] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:21.705250 22932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:21.705380 22932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [term 1 LEADER]: Becoming Leader. State: Replica: 81e66b0c616b45ee90bb471f4e62fb58, State: Running, Role: LEADER
I20260923 15:11:21.705476 22927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:21.705468 22932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66b0c616b45ee90bb471f4e62fb58" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34605 } }
I20260923 15:11:21.706231 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81e66b0c616b45ee90bb471f4e62fb58. Latest consensus state: current_term: 1 leader_uuid: "81e66b0c616b45ee90bb471f4e62fb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66b0c616b45ee90bb471f4e62fb58" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34605 } } }
I20260923 15:11:21.706332 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:21.706548 22933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81e66b0c616b45ee90bb471f4e62fb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81e66b0c616b45ee90bb471f4e62fb58" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34605 } } }
I20260923 15:11:21.706638 22933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:21.706715 22941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:21.707228 22941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:21.709049 22941 catalog_manager.cc:1347] Generated new cluster ID: 437e3f2c12b44a9f8ffe48c1e972649b
I20260923 15:11:21.709120 22941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:11:21 dist-test-slave-t5lt krb5kdc[22846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:21.713835 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:21.702955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.21.204.126:34605)
0923 15:11:21.703140 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:21.703148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:21.703176 (+ 28us) server_negotiation.cc:407] Connection header received
0923 15:11:21.703360 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:21.703365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:21.703418 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:21.703532 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:21.704350 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:21.705450 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:21.706612 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:21.707229 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:21.707661 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:21.710125 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:21.710129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:21.710130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:21.710142 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:21.712100 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:21.712479 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:21.712484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:21.712484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:21.712613 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:21.712812 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:21.712813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:21.712813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:21.713128 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:21.713303 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:21.713373 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 15:11:21.713614 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260923 15:11:21.725471 22941 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:21.726064 22941 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:21.733741 22941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81e66b0c616b45ee90bb471f4e62fb58: Generated new TSK 0
I20260923 15:11:21.733978 22941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:21.766120 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34605
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:21.851030 22953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:21.851238 22953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:21.851263 22953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:21.852353 22953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:21.852393 22953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:21.852421 22953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:21.853900 22953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:21.854101 22953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:21.854276 22953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:21.859493 22961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:21.859606 22953 server_base.cc:1034] running on GCE node
W20260923 15:11:21.859493 22959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:21.859519 22958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:21.859959 22953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:21.860209 22953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:21.861375 22953 hybrid_clock.cc:648] HybridClock initialized: now 1790176281861355 us; error 44 us; skew 500 ppm
Sep 23 15:11:21 dist-test-slave-t5lt krb5kdc[22846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:21.864192 22953 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:21.864710 22953 webserver.cc:460] Webserver started at http://127.21.204.65:32787/ using document root <none> and password file <none>
I20260923 15:11:21.864953 22953 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:21.865025 22953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:21.865209 22953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:21.866322 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "340248bdfa924e38a65a621fd45292e5"
format_stamp: "Formatted at 2026-09-23 15:11:21 on dist-test-slave-t5lt"
server_key: "be9e955127e711cd20552a164371b7a1"
server_key_iv: "52da41402ae9b878c25ddb87503b1122"
server_key_version: "encryptionkey@0"
I20260923 15:11:21.866725 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "340248bdfa924e38a65a621fd45292e5"
format_stamp: "Formatted at 2026-09-23 15:11:21 on dist-test-slave-t5lt"
server_key: "be9e955127e711cd20552a164371b7a1"
server_key_iv: "52da41402ae9b878c25ddb87503b1122"
server_key_version: "encryptionkey@0"
I20260923 15:11:21.868155 22953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:11:21.869038 22968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:21.869309 22953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:21.869426 22953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "340248bdfa924e38a65a621fd45292e5"
format_stamp: "Formatted at 2026-09-23 15:11:21 on dist-test-slave-t5lt"
server_key: "be9e955127e711cd20552a164371b7a1"
server_key_iv: "52da41402ae9b878c25ddb87503b1122"
server_key_version: "encryptionkey@0"
I20260923 15:11:21.869493 22953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:21.896721 22953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:21.899057 22953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:21.899252 22953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:21.899511 22953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:21.899852 22953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:21.899916 22953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:21.899959 22953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:21.899989 22953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:21.906149 22953 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:36447
I20260923 15:11:21.906222 23081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:36447 every 8 connection(s)
I20260923 15:11:21.906525 22953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:21.911773 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 22953
I20260923 15:11:21.911892 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:21.912067 22321 external_mini_cluster.cc:1428] Setting key 94b4bf7b0dcd3be70a7f003c695b9d8b
Sep 23 15:11:21 dist-test-slave-t5lt krb5kdc[22846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176281, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:21.919787 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:21.907272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34605 (local address 127.21.204.65:48985)
0923 15:11:21.907541 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:21.907621 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:21.908569 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:21.908805 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:21.908809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:21.909074 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:21.909286 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:21.909291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:21.910218 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:21.910219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:21.911757 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:21.911761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:21.912394 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:21.912395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:21.912537 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:21.913160 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:21.913170 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:21.915254 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:21.917587 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:21.917595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:21.917598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:21.918101 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:21.918328 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:21.918328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:21.918329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:21.918477 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:21.918810 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:21.918817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:21.919042 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 15:11:21.919265 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 15:11:21.919512 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":104,"threads_started":1}
I20260923 15:11:21.920295 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:21.907288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:48985 (local address 127.21.204.126:34605)
0923 15:11:21.907367 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:21.907370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:21.907754 (+ 384us) server_negotiation.cc:407] Connection header received
0923 15:11:21.908672 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:21.908673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:21.908689 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:21.908747 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:21.909399 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:21.910077 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:21.911867 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:21.912251 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:21.912320 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:21.915427 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:21.915429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:21.915429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:21.915440 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:21.917481 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:21.918191 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:21.918193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:21.918193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:21.918254 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:21.918545 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:21.918547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:21.918547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:21.918716 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:21.918809 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:21.920081 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0923 15:11:21.920177 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 15:11:21.920873 23082 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34605
I20260923 15:11:21.921032 23082 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:21.921250 23082 heartbeater.cc:511] Master 127.21.204.126:34605 requested a full tablet report, sending...
I20260923 15:11:21.921814 22891 ts_manager.cc:194] Registered new tserver with Master: 340248bdfa924e38a65a621fd45292e5 (127.21.204.65:36447)
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:11:21.922567 22891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:48985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:21.941581 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34605
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:22.023249 23087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:22.023459 23087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:22.023483 23087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:22.024634 23087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:22.024677 23087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:22.024705 23087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:22.026213 23087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:22.026454 23087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:22.026710 23087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:22.031816 23093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:22.031890 23092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:22.032003 23095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:22.032135 23087 server_base.cc:1034] running on GCE node
I20260923 15:11:22.032354 23087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:22.032629 23087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:22.033797 23087 hybrid_clock.cc:648] HybridClock initialized: now 1790176282033787 us; error 30 us; skew 500 ppm
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:22.036288 23087 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:22.036731 23087 webserver.cc:460] Webserver started at http://127.21.204.66:43171/ using document root <none> and password file <none>
I20260923 15:11:22.036912 23087 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:22.036952 23087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:22.037037 23087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:22.038090 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "c85fe4e03e684699902c5f739dd2b1ea"
format_stamp: "Formatted at 2026-09-23 15:11:22 on dist-test-slave-t5lt"
server_key: "a921e441f2c3c9b55aa87949d5beaf35"
server_key_iv: "2818bc02443631f545778316d614dbf1"
server_key_version: "encryptionkey@0"
I20260923 15:11:22.038399 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "c85fe4e03e684699902c5f739dd2b1ea"
format_stamp: "Formatted at 2026-09-23 15:11:22 on dist-test-slave-t5lt"
server_key: "a921e441f2c3c9b55aa87949d5beaf35"
server_key_iv: "2818bc02443631f545778316d614dbf1"
server_key_version: "encryptionkey@0"
I20260923 15:11:22.039676 23087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:22.040433 23102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:22.040630 23087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:22.040750 23087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "c85fe4e03e684699902c5f739dd2b1ea"
format_stamp: "Formatted at 2026-09-23 15:11:22 on dist-test-slave-t5lt"
server_key: "a921e441f2c3c9b55aa87949d5beaf35"
server_key_iv: "2818bc02443631f545778316d614dbf1"
server_key_version: "encryptionkey@0"
I20260923 15:11:22.040825 23087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:22.055317 23087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:22.057281 23087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:22.057437 23087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:22.057677 23087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:22.058041 23087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:22.058096 23087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:22.058145 23087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:22.058173 23087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:22.064116 23087 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:36165
I20260923 15:11:22.064162 23215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:36165 every 8 connection(s)
I20260923 15:11:22.064469 23087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:22.066542 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 23087
I20260923 15:11:22.066646 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:22.066816 22321 external_mini_cluster.cc:1428] Setting key 830bce6bd8e9e39f70825363ff94851f
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:22.076100 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.065243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:45873 (local address 127.21.204.126:34605)
0923 15:11:22.065311 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.065313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.065510 (+ 197us) server_negotiation.cc:407] Connection header received
0923 15:11:22.066505 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.066506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.066523 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.066603 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:22.067677 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.068442 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.069857 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.070094 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.070158 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.072694 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:22.072697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:22.072697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:22.072706 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:22.074605 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.075022 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.075023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.075023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.075090 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.075392 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.075393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.075393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.075549 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:22.075629 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.075909 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.076001 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 15:11:22.076212 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.065238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34605 (local address 127.21.204.66:45873)
0923 15:11:22.065387 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:22.065428 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:22.066402 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:22.066741 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:22.066746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:22.067053 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:22.067371 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.067378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.068538 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.068539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.069764 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.069768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.070182 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.070200 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.070344 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.070845 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:22.070855 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:22.072517 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:22.074710 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:22.074717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:22.074719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:22.074943 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:22.075194 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:22.075195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:22.075196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:22.075304 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:22.075639 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:22.075646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:22.075764 (+ 118us) client_negotiation.cc:769] Sending connection context
0923 15:11:22.075859 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 15:11:22.076005 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
I20260923 15:11:22.076874 23216 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34605
I20260923 15:11:22.076959 23216 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:22.077104 23216 heartbeater.cc:511] Master 127.21.204.126:34605 requested a full tablet report, sending...
I20260923 15:11:22.077541 22891 ts_manager.cc:194] Registered new tserver with Master: c85fe4e03e684699902c5f739dd2b1ea (127.21.204.66:36165)
I20260923 15:11:22.078044 22891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:45873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:22.097283 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34605
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:22.181027 23221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:22.181283 23221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:22.181313 23221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:22.182483 23221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:22.182524 23221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:22.182554 23221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:22.184141 23221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:22.184342 23221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:22.184537 23221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:11:22.190054 23221 server_base.cc:1034] running on GCE node
W20260923 15:11:22.190160 23229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:22.190093 23227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:22.190140 23226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:22.190486 23221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:22.190745 23221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:22.191911 23221 hybrid_clock.cc:648] HybridClock initialized: now 1790176282191893 us; error 42 us; skew 500 ppm
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:22.195000 23221 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:22.195480 23221 webserver.cc:460] Webserver started at http://127.21.204.67:34301/ using document root <none> and password file <none>
I20260923 15:11:22.195724 23221 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:22.195791 23221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:22.195935 23221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:22.197063 23221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "77de2a30a87d4dac88fcef206d7d6471"
format_stamp: "Formatted at 2026-09-23 15:11:22 on dist-test-slave-t5lt"
server_key: "73c9420842a3ae882c9b02be30cfb37f"
server_key_iv: "6d15137ffdce14241717116bcb664643"
server_key_version: "encryptionkey@0"
I20260923 15:11:22.197415 23221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "77de2a30a87d4dac88fcef206d7d6471"
format_stamp: "Formatted at 2026-09-23 15:11:22 on dist-test-slave-t5lt"
server_key: "73c9420842a3ae882c9b02be30cfb37f"
server_key_iv: "6d15137ffdce14241717116bcb664643"
server_key_version: "encryptionkey@0"
I20260923 15:11:22.198840 23221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:22.199607 23236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:22.199800 23221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:22.199900 23221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "77de2a30a87d4dac88fcef206d7d6471"
format_stamp: "Formatted at 2026-09-23 15:11:22 on dist-test-slave-t5lt"
server_key: "73c9420842a3ae882c9b02be30cfb37f"
server_key_iv: "6d15137ffdce14241717116bcb664643"
server_key_version: "encryptionkey@0"
I20260923 15:11:22.199980 23221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:22.213490 23221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:22.215780 23221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:22.215938 23221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:22.216162 23221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:22.216459 23221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:22.216511 23221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:22.216552 23221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:22.216603 23221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:22.226501 23221 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:42345
I20260923 15:11:22.226549 23349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:42345 every 8 connection(s)
I20260923 15:11:22.226961 23221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:22.232677 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 23221
I20260923 15:11:22.232810 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:22.233002 22321 external_mini_cluster.cc:1428] Setting key 59e36822688984a206b128941ae59955
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:22.237056 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.227724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:35059 (local address 127.21.204.126:34605)
0923 15:11:22.227794 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.227796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.227965 (+ 169us) server_negotiation.cc:407] Connection header received
0923 15:11:22.229036 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.229037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.229048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.229104 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:22.229837 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.230411 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.231486 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.231697 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.231764 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.234542 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:22.234544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:22.234544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:22.234552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:22.235788 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.236250 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.236252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.236252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.236313 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.236533 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.236534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.236534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.236641 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:22.236699 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.236904 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.236993 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 15:11:22.237300 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.227689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34605 (local address 127.21.204.67:35059)
0923 15:11:22.227843 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:22.227883 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:22.228932 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:22.229162 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:22.229166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:22.229456 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:22.229749 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.229754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.230518 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.230519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.231383 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.231385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.231780 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.231781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.231884 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.232417 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:22.232425 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:22.234447 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:22.235910 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:22.235922 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:22.235924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:22.236168 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:22.236378 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:22.236379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:22.236379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:22.236475 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:22.236698 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:22.236703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:22.236810 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 15:11:22.236889 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 15:11:22.237056 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260923 15:11:22.237803 23350 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34605
I20260923 15:11:22.237919 23350 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:22.238090 23350 heartbeater.cc:511] Master 127.21.204.126:34605 requested a full tablet report, sending...
I20260923 15:11:22.238443 22891 ts_manager.cc:194] Registered new tserver with Master: 77de2a30a87d4dac88fcef206d7d6471 (127.21.204.67:42345)
I20260923 15:11:22.238842 22891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:35059
I20260923 15:11:22.245438 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:11:22.253090 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.247074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.21.204.126:34605)
0923 15:11:22.247151 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.247152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.247263 (+ 111us) server_negotiation.cc:407] Connection header received
0923 15:11:22.247304 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.247304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.247313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.247362 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:22.247949 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.248415 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.249120 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.249316 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.249374 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.250707 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:22.250708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:22.250708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:22.250714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:22.252121 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.252423 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.252424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.252424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.252500 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.252667 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.252667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.252667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.252796 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:22.252872 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.252951 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.253004 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 15:11:22.255115 22891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:22.256698 22891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:22.263510 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.259134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36447 (local address 127.0.0.1:57238)
0923 15:11:22.259289 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:22.259329 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:22.259389 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:22.260014 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:22.260015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:22.260017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:22.260053 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.260054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.261389 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.261390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.262454 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.262456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.263190 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.263192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.263275 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.263288 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 15:11:22.263358 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 15:11:22.263411 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260923 15:11:22.263705 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.259638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.21.204.65:36447)
0923 15:11:22.259854 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.259859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.259871 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:11:22.259913 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.259915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.259931 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.260017 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:22.260151 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.261284 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.262545 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.263099 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.263191 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.263250 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.263559 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.263604 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":147,"threads_started":1}
I20260923 15:11:22.263770 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.259327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:36165 (local address 127.0.0.1:41312)
0923 15:11:22.259532 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:22.259542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:22.259596 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:22.260357 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:22.260358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:22.260360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:22.260403 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.260404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.261678 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.261679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.262596 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.262599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.263538 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.263539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.263598 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.263603 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:11:22.263652 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 15:11:22.263691 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":158,"threads_started":1}
I20260923 15:11:22.263775 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.259134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:42345 (local address 127.0.0.1:57076)
0923 15:11:22.259290 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:22.259328 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:22.259409 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:22.259693 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:22.259696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:22.259699 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:22.259879 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.259882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.261065 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.261066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.262336 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.262339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.263351 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.263352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.263405 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.263408 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:22.263460 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 15:11:22.263498 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260923 15:11:22.264299 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.259978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.21.204.66:36165)
0923 15:11:22.260187 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.260191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.260200 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:22.260240 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.260241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.260256 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.260358 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:22.260482 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.261483 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.262703 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.263441 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.264005 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.264069 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.264095 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.264139 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260923 15:11:22.264604 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.259232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.21.204.67:42345)
0923 15:11:22.259515 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.259519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.259526 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:22.259557 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.259558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.259570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.259667 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:22.260032 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.260960 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.262423 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.263262 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.263352 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.263407 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.263489 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.263540 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":90,"threads_started":1}
I20260923 15:11:22.264580 23016 tablet_service.cc:1467] Processing CreateTablet for tablet f31dd7ef1e7b420a85336d90e610eb5e (DEFAULT_TABLE table=test-table [id=f52196af81404a39ac27cca86cfadaaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:22.264968 23016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31dd7ef1e7b420a85336d90e610eb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:22.265018 23150 tablet_service.cc:1467] Processing CreateTablet for tablet f31dd7ef1e7b420a85336d90e610eb5e (DEFAULT_TABLE table=test-table [id=f52196af81404a39ac27cca86cfadaaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:22.265345 23150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31dd7ef1e7b420a85336d90e610eb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:22.265364 23284 tablet_service.cc:1467] Processing CreateTablet for tablet f31dd7ef1e7b420a85336d90e610eb5e (DEFAULT_TABLE table=test-table [id=f52196af81404a39ac27cca86cfadaaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:22.265697 23284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31dd7ef1e7b420a85336d90e610eb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:22.268491 23367 tablet_bootstrap.cc:492] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471: Bootstrap starting.
I20260923 15:11:22.269270 23368 tablet_bootstrap.cc:492] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea: Bootstrap starting.
I20260923 15:11:22.269491 23369 tablet_bootstrap.cc:492] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5: Bootstrap starting.
I20260923 15:11:22.269821 23367 tablet_bootstrap.cc:654] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:22.270125 23368 tablet_bootstrap.cc:654] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:22.270293 23369 tablet_bootstrap.cc:654] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:22.270673 23367 log.cc:824] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:22.270783 23368 log.cc:824] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:22.270790 23369 log.cc:824] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:22.271569 23367 tablet_bootstrap.cc:492] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471: No bootstrap required, opened a new log
I20260923 15:11:22.271657 23367 ts_tablet_manager.cc:1397] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:11:22.271768 23369 tablet_bootstrap.cc:492] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5: No bootstrap required, opened a new log
I20260923 15:11:22.271843 23369 ts_tablet_manager.cc:1397] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 15:11:22.271955 23368 tablet_bootstrap.cc:492] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea: No bootstrap required, opened a new log
I20260923 15:11:22.272048 23368 ts_tablet_manager.cc:1397] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:11:22.273825 23367 raft_consensus.cc:348] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.274053 23367 raft_consensus.cc:374] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:22.274102 23367 raft_consensus.cc:729] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77de2a30a87d4dac88fcef206d7d6471, State: Initialized, Role: FOLLOWER
I20260923 15:11:22.274221 23367 consensus_queue.cc:260] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.274546 23367 ts_tablet_manager.cc:1428] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:22.274574 23350 heartbeater.cc:503] Master 127.21.204.126:34605 was elected leader, sending a full tablet report...
I20260923 15:11:22.274490 23368 raft_consensus.cc:348] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.274639 23368 raft_consensus.cc:374] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:22.274682 23368 raft_consensus.cc:729] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85fe4e03e684699902c5f739dd2b1ea, State: Initialized, Role: FOLLOWER
I20260923 15:11:22.274793 23368 consensus_queue.cc:260] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.274413 23369 raft_consensus.cc:348] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.274916 23369 raft_consensus.cc:374] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:22.274971 23369 raft_consensus.cc:729] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340248bdfa924e38a65a621fd45292e5, State: Initialized, Role: FOLLOWER
I20260923 15:11:22.275101 23368 ts_tablet_manager.cc:1428] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:22.275103 23369 consensus_queue.cc:260] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.275146 23216 heartbeater.cc:503] Master 127.21.204.126:34605 was elected leader, sending a full tablet report...
I20260923 15:11:22.275394 23082 heartbeater.cc:503] Master 127.21.204.126:34605 was elected leader, sending a full tablet report...
I20260923 15:11:22.275394 23369 ts_tablet_manager.cc:1428] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260923 15:11:22.315045 23217 tablet.cc:2369] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:22.407316 23083 tablet.cc:2369] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:22.477682 23351 tablet.cc:2369] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:22.479089 23373 raft_consensus.cc:482] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:22.479187 23373 raft_consensus.cc:504] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.479497 23373 leader_election.cc:288] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 340248bdfa924e38a65a621fd45292e5 (127.21.204.65:36447), c85fe4e03e684699902c5f739dd2b1ea (127.21.204.66:36165)
I20260923 15:11:22.482964 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.479898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:34653 (local address 127.21.204.65:36447)
0923 15:11:22.479981 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.479984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.480104 (+ 120us) server_negotiation.cc:407] Connection header received
0923 15:11:22.480187 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.480189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.480202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.480265 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:22.480531 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.481154 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.482241 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.482604 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.482662 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.482669 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.482842 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.482880 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 15:11:22.482940 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.479784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36447 (local address 127.21.204.67:34653)
0923 15:11:22.480019 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:22.480032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:22.480103 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:22.480287 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:22.480288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:22.480290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:22.480395 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.480398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.481237 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.481238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.482144 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.482147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.482684 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.482685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.482765 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.482769 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:22.482830 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 15:11:22.482874 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260923 15:11:22.483103 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.480133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:36165 (local address 127.21.204.67:33151)
0923 15:11:22.480324 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:22.480333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:22.480386 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:22.480517 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:22.480518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:22.480520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:22.480593 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.480594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.481350 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.481351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.482177 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:22.482179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.482834 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.482835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:22.482908 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.482913 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:11:22.482969 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 15:11:22.483015 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":111,"threads_started":1}
I20260923 15:11:22.483137 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.480222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:33151 (local address 127.21.204.66:36165)
0923 15:11:22.480295 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.480297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.480393 (+ 96us) server_negotiation.cc:407] Connection header received
0923 15:11:22.480427 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.480428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.480440 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.480505 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:22.480681 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.481249 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.482289 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.482725 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.482808 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.482818 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.483004 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.483048 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 15:11:22.483316 23036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31dd7ef1e7b420a85336d90e610eb5e" candidate_uuid: "77de2a30a87d4dac88fcef206d7d6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340248bdfa924e38a65a621fd45292e5" is_pre_election: true
I20260923 15:11:22.483438 23036 raft_consensus.cc:2454] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77de2a30a87d4dac88fcef206d7d6471 in term 0.
I20260923 15:11:22.483677 23170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31dd7ef1e7b420a85336d90e610eb5e" candidate_uuid: "77de2a30a87d4dac88fcef206d7d6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c85fe4e03e684699902c5f739dd2b1ea" is_pre_election: true
I20260923 15:11:22.483824 23170 raft_consensus.cc:2454] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77de2a30a87d4dac88fcef206d7d6471 in term 0.
I20260923 15:11:22.484086 23239 leader_election.cc:302] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340248bdfa924e38a65a621fd45292e5, 77de2a30a87d4dac88fcef206d7d6471; no voters:
I20260923 15:11:22.484227 23373 raft_consensus.cc:2781] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:22.484321 23373 raft_consensus.cc:482] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:22.484359 23373 raft_consensus.cc:3037] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:22.485163 23373 raft_consensus.cc:504] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.485319 23373 leader_election.cc:288] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [CANDIDATE]: Term 1 election: Requested vote from peers 340248bdfa924e38a65a621fd45292e5 (127.21.204.65:36447), c85fe4e03e684699902c5f739dd2b1ea (127.21.204.66:36165)
I20260923 15:11:22.485510 23036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31dd7ef1e7b420a85336d90e610eb5e" candidate_uuid: "77de2a30a87d4dac88fcef206d7d6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340248bdfa924e38a65a621fd45292e5"
I20260923 15:11:22.485545 23170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31dd7ef1e7b420a85336d90e610eb5e" candidate_uuid: "77de2a30a87d4dac88fcef206d7d6471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c85fe4e03e684699902c5f739dd2b1ea"
I20260923 15:11:22.485632 23170 raft_consensus.cc:3037] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:22.485630 23036 raft_consensus.cc:3037] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:22.486486 23170 raft_consensus.cc:2454] T f31dd7ef1e7b420a85336d90e610eb5e P c85fe4e03e684699902c5f739dd2b1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77de2a30a87d4dac88fcef206d7d6471 in term 1.
I20260923 15:11:22.486498 23036 raft_consensus.cc:2454] T f31dd7ef1e7b420a85336d90e610eb5e P 340248bdfa924e38a65a621fd45292e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77de2a30a87d4dac88fcef206d7d6471 in term 1.
I20260923 15:11:22.486706 23239 leader_election.cc:302] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340248bdfa924e38a65a621fd45292e5, 77de2a30a87d4dac88fcef206d7d6471; no voters:
I20260923 15:11:22.486804 23373 raft_consensus.cc:2781] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:22.486958 23373 raft_consensus.cc:686] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [term 1 LEADER]: Becoming Leader. State: Replica: 77de2a30a87d4dac88fcef206d7d6471, State: Running, Role: LEADER
I20260923 15:11:22.487072 23373 consensus_queue.cc:237] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } }
I20260923 15:11:22.487962 22891 catalog_manager.cc:5495] T f31dd7ef1e7b420a85336d90e610eb5e P 77de2a30a87d4dac88fcef206d7d6471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77de2a30a87d4dac88fcef206d7d6471 (127.21.204.67). New cstate: current_term: 1 leader_uuid: "77de2a30a87d4dac88fcef206d7d6471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77de2a30a87d4dac88fcef206d7d6471" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "340248bdfa924e38a65a621fd45292e5" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c85fe4e03e684699902c5f739dd2b1ea" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36165 } health_report { overall_health: UNKNOWN } } }
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:22.499452 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.491258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.21.204.65:36447)
0923 15:11:22.491339 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.491342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.491469 (+ 127us) server_negotiation.cc:407] Connection header received
0923 15:11:22.491571 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.491573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.491584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.491647 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:22.492249 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.492773 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.493592 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.493806 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.493911 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.496443 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:22.496455 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:22.496455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:22.496463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:22.498236 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.498628 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.498631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.498632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.498701 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.498908 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.498909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.498909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.499062 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:22.499221 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.499263 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.499344 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:22.523960 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.516705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.21.204.65:36447)
0923 15:11:22.516759 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.516761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.516818 (+ 57us) server_negotiation.cc:407] Connection header received
0923 15:11:22.516869 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.516870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.516880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.516930 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:22.517510 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.518205 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.518950 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.519205 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.519268 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.521602 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:22.521603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:22.521604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:22.521610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:22.522899 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.523236 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.523237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.523238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.523289 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.523483 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.523484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.523484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.523590 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:22.523678 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.523820 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.523888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[22846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:22.546166 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:22.538931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.21.204.65:36447)
0923 15:11:22.538997 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:22.538999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:22.539063 (+ 64us) server_negotiation.cc:407] Connection header received
0923 15:11:22.539169 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:22.539170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:22.539180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:22.539237 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:22.539794 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.540300 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.541016 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:22.541221 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:22.541292 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:22.543718 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:22.543720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:22.543720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:22.543726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:22.545170 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.545498 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.545499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.545499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.545547 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:22.545701 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:22.545702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:22.545703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:22.545788 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:22.545859 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:22.546033 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 15:11:22.546101 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 15:11:22.546722 22996 tablet_service.cc:2908] Rejecting scan request for tablet f31dd7ef1e7b420a85336d90e610eb5e: Uninitialized: clean time has not yet been initialized
I20260923 15:11:22.547585 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 22953
I20260923 15:11:22.553129 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 23087
I20260923 15:11:22.557552 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 23221
I20260923 15:11:22.562091 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 22859
2026-09-23T15:11:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2173 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[23391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[23391](info): set up 2 sockets
Sep 23 15:11:22 dist-test-slave-t5lt krb5kdc[23391](info): commencing operation
krb5kdc: starting...
W20260923 15:11:23.598841 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:23 dist-test-slave-t5lt krb5kdc[23391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:23.717031 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33379
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:33379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:23.797632 23404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:23.797878 23404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:23.797946 23404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:23.799140 23404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:23.799207 23404 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 15:11:23.799228 23404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:23.799249 23404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:23.799283 23404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:23.799306 23404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:23.800803 23404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:33379
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:23.801082 23404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:23.801314 23404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:11:23.806864 23404 server_base.cc:1034] running on GCE node
W20260923 15:11:23.806846 23410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:23.806847 23409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:23.806883 23412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:23.807389 23404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:23.807700 23404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:23.808890 23404 hybrid_clock.cc:648] HybridClock initialized: now 1790176283808855 us; error 48 us; skew 500 ppm
Sep 23 15:11:23 dist-test-slave-t5lt krb5kdc[23391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176283, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:23.811673 23404 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:23.812141 23404 webserver.cc:460] Webserver started at http://127.21.204.126:36503/ using document root <none> and password file <none>
I20260923 15:11:23.812383 23404 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:23.812430 23404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:23.812584 23404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:23.813639 23404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "d25d2d852e004a0298931e88579ca6a3"
format_stamp: "Formatted at 2026-09-23 15:11:23 on dist-test-slave-t5lt"
server_key: "6efd33d34b590c349d12dcfddfd0dae1"
server_key_iv: "dbd830dbf955c877fc48d629a895a350"
server_key_version: "encryptionkey@0"
I20260923 15:11:23.814052 23404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "d25d2d852e004a0298931e88579ca6a3"
format_stamp: "Formatted at 2026-09-23 15:11:23 on dist-test-slave-t5lt"
server_key: "6efd33d34b590c349d12dcfddfd0dae1"
server_key_iv: "dbd830dbf955c877fc48d629a895a350"
server_key_version: "encryptionkey@0"
I20260923 15:11:23.815423 23404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:23.816248 23419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:23.816483 23404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:23.816558 23404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "d25d2d852e004a0298931e88579ca6a3"
format_stamp: "Formatted at 2026-09-23 15:11:23 on dist-test-slave-t5lt"
server_key: "6efd33d34b590c349d12dcfddfd0dae1"
server_key_iv: "dbd830dbf955c877fc48d629a895a350"
server_key_version: "encryptionkey@0"
I20260923 15:11:23.816656 23404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:23.828997 23404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:23.831136 23404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:23.831286 23404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:23.835915 23404 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:33379
I20260923 15:11:23.836134 23481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:33379 every 8 connection(s)
I20260923 15:11:23.836369 23404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:23.837015 23482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:23.839365 23482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3: Bootstrap starting.
I20260923 15:11:23.839903 23482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:23.840140 23482 log.cc:824] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:23.840788 23482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3: No bootstrap required, opened a new log
I20260923 15:11:23.841754 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 23404
I20260923 15:11:23.841863 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:23.842082 22321 external_mini_cluster.cc:1428] Setting key 44d719f96173261eb738f6d7f5faf0cb
I20260923 15:11:23.843410 23482 raft_consensus.cc:348] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d2d852e004a0298931e88579ca6a3" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33379 } }
I20260923 15:11:23.843585 23482 raft_consensus.cc:374] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:23.843624 23482 raft_consensus.cc:729] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25d2d852e004a0298931e88579ca6a3, State: Initialized, Role: FOLLOWER
I20260923 15:11:23.843781 23482 consensus_queue.cc:260] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d2d852e004a0298931e88579ca6a3" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33379 } }
I20260923 15:11:23.843863 23482 raft_consensus.cc:388] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:23.843886 23482 raft_consensus.cc:482] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:23.843950 23482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:23.844591 23482 raft_consensus.cc:504] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d2d852e004a0298931e88579ca6a3" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33379 } }
I20260923 15:11:23.844698 23482 leader_election.cc:302] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d25d2d852e004a0298931e88579ca6a3; no voters:
I20260923 15:11:23.844854 23482 leader_election.cc:288] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:23.844923 23487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:23.845095 23487 raft_consensus.cc:686] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [term 1 LEADER]: Becoming Leader. State: Replica: d25d2d852e004a0298931e88579ca6a3, State: Running, Role: LEADER
I20260923 15:11:23.845101 23482 sys_catalog.cc:564] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:23.845201 23487 consensus_queue.cc:237] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d2d852e004a0298931e88579ca6a3" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33379 } }
I20260923 15:11:23.845634 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25d2d852e004a0298931e88579ca6a3. Latest consensus state: current_term: 1 leader_uuid: "d25d2d852e004a0298931e88579ca6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d2d852e004a0298931e88579ca6a3" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33379 } } }
I20260923 15:11:23.845793 23488 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d25d2d852e004a0298931e88579ca6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25d2d852e004a0298931e88579ca6a3" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33379 } } }
I20260923 15:11:23.845970 23488 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:23.846045 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:23.846351 23499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:23.846916 23499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:23.849064 23499 catalog_manager.cc:1347] Generated new cluster ID: bcced9aa0cf24d1a9b13513e45b46335
I20260923 15:11:23.849131 23499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:11:23 dist-test-slave-t5lt krb5kdc[23391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:23.853904 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:23.842595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.21.204.126:33379)
0923 15:11:23.842825 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:23.842837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:23.842878 (+ 41us) server_negotiation.cc:407] Connection header received
0923 15:11:23.843365 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:23.843375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:23.843439 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:23.843601 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:23.844329 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:23.845746 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:23.846943 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:23.847497 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:23.847613 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:23.850256 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:23.850260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:23.850260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:23.850272 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:23.852180 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:23.852670 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:23.852673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:23.852674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:23.852796 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:23.853009 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:23.853010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:23.853011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:23.853295 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:23.853441 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:23.853560 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 15:11:23.853691 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260923 15:11:23.878418 23499 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:23.879021 23499 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:23.891181 23499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3: Generated new TSK 0
I20260923 15:11:23.891532 23499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:23.921720 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33379
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:24.004794 23508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:24.004998 23508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:24.005024 23508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:24.006517 23508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:24.006572 23508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:24.006594 23508 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:24.006613 23508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:24.008157 23508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:33379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:24.008358 23508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:24.008538 23508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:24.013733 23514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:24.013722 23513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:24.013753 23508 server_base.cc:1034] running on GCE node
W20260923 15:11:24.013722 23516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:24.014194 23508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:24.014436 23508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:24.015615 23508 hybrid_clock.cc:648] HybridClock initialized: now 1790176284015600 us; error 31 us; skew 500 ppm
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:24.018150 23508 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:24.018565 23508 webserver.cc:460] Webserver started at http://127.21.204.65:35519/ using document root <none> and password file <none>
I20260923 15:11:24.018821 23508 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:24.018893 23508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:24.019104 23508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:24.020234 23508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "44f837811ea549359a0e45a97e180b57"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "4ba7117b18e77f700fbda162a4f7b869"
server_key_iv: "a3289c3abaf496da5749c72c11e5b7b9"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.020653 23508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "44f837811ea549359a0e45a97e180b57"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "4ba7117b18e77f700fbda162a4f7b869"
server_key_iv: "a3289c3abaf496da5749c72c11e5b7b9"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.022060 23508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:11:24.022807 23523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.022972 23508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:24.023070 23508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "44f837811ea549359a0e45a97e180b57"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "4ba7117b18e77f700fbda162a4f7b869"
server_key_iv: "a3289c3abaf496da5749c72c11e5b7b9"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.023164 23508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:24.035861 23508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:24.037793 23508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:24.038002 23508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:24.038501 23508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:24.038559 23508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.038614 23508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:24.038664 23508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.045614 23508 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:43951
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:24.045681 23638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:43951 every 8 connection(s)
I20260923 15:11:24.046167 23508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:24.046921 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 23508
I20260923 15:11:24.047042 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:24.047202 22321 external_mini_cluster.cc:1428] Setting key 618d3b5132cd555a25978b488edd9243
I20260923 15:11:24.050040 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.039109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:38407 (local address 127.21.204.126:33379)
0923 15:11:24.039157 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.039158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.039366 (+ 208us) server_negotiation.cc:407] Connection header received
0923 15:11:24.039959 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.039961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.039975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.040051 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.041040 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.041921 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.043043 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.043388 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.043484 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.046269 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.046271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.046271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.046284 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.048228 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.048693 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.048695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.048695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.048770 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.049102 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.049103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.049104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.049309 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.049386 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.049776 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.049942 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:11:24.050192 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.039080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33379 (local address 127.21.204.65:38407)
0923 15:11:24.039250 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.039284 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.039807 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.040192 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.040197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.040614 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:24.040918 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.040924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.042037 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.042038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.042919 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.042922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.043494 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.043495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.043686 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.044274 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:24.044286 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:24.046169 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:24.048337 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.048345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.048348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.048594 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.048856 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.048857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.048858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.048983 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.049399 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:24.049406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:24.049582 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.049716 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.049931 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260923 15:11:24.050956 23639 heartbeater.cc:347] Connected to a master server at 127.21.204.126:33379
I20260923 15:11:24.051045 23639 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:24.051241 23639 heartbeater.cc:511] Master 127.21.204.126:33379 requested a full tablet report, sending...
I20260923 15:11:24.051834 23436 ts_manager.cc:194] Registered new tserver with Master: 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951)
I20260923 15:11:24.052807 23436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:38407
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:11:24.058312 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.052120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:60349 (local address 127.21.204.126:33379)
0923 15:11:24.052160 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.052162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.052295 (+ 133us) server_negotiation.cc:407] Connection header received
0923 15:11:24.052367 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.052368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.052389 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.052434 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.052750 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.053260 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.054033 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.054352 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.054450 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.055485 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.055487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.055487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.055495 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.057166 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.057521 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.057522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.057522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.057582 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.057823 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.057824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.057825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.057992 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.058066 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.058185 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.058253 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:11:24.058319 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.052065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33379 (local address 127.21.204.65:60349)
0923 15:11:24.052215 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.052225 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.052309 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.052436 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.052437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.052622 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:24.052672 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.052674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.053356 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.053356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.053957 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.053960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.054478 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.054479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.054549 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.055076 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:24.055083 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:24.055380 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:24.057261 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.057262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.057262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.057428 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.057659 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.057660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.057660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.057755 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.058073 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:24.058074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:24.058113 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.058169 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.058247 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:24.078701 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33379
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:24.161624 23648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:24.161825 23648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:24.161849 23648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:24.163029 23648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:24.163103 23648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:24.163138 23648 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:24.163177 23648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:24.164651 23648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:33379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:24.164922 23648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:24.165166 23648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:24.170264 23653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:24.170390 23648 server_base.cc:1034] running on GCE node
W20260923 15:11:24.170281 23654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:24.170217 23656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:24.170783 23648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:24.171033 23648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:24.172222 23648 hybrid_clock.cc:648] HybridClock initialized: now 1790176284172204 us; error 30 us; skew 500 ppm
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:24.174783 23648 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:24.175277 23648 webserver.cc:460] Webserver started at http://127.21.204.66:36481/ using document root <none> and password file <none>
I20260923 15:11:24.175526 23648 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:24.175597 23648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:24.175746 23648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:24.176769 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "26c3d1712d11414e9e263561ee53c9e1"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "cf0dc1b0780b93c1e248ac702ec465bb"
server_key_iv: "3292ac3059098160a9a8117f8a8f4577"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.177134 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "26c3d1712d11414e9e263561ee53c9e1"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "cf0dc1b0780b93c1e248ac702ec465bb"
server_key_iv: "3292ac3059098160a9a8117f8a8f4577"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.178613 23648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:24.179342 23663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.179749 23648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:24.179880 23648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "26c3d1712d11414e9e263561ee53c9e1"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "cf0dc1b0780b93c1e248ac702ec465bb"
server_key_iv: "3292ac3059098160a9a8117f8a8f4577"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.179963 23648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:24.191444 23648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:24.193430 23648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:24.193603 23648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:24.194229 23648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:24.194295 23648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.194329 23648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:24.194356 23648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:24.201596 23648 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:35981
I20260923 15:11:24.201681 23778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:35981 every 8 connection(s)
I20260923 15:11:24.202042 23648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:24.203939 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 23648
I20260923 15:11:24.204077 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:24.204262 22321 external_mini_cluster.cc:1428] Setting key e527eb9a5221b9ebc862865a04ee4f91
I20260923 15:11:24.204548 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.194868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:45207 (local address 127.21.204.126:33379)
0923 15:11:24.194955 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.194957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.194970 (+ 13us) server_negotiation.cc:407] Connection header received
0923 15:11:24.195381 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.195383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.195396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.195485 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.196172 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.196819 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.198574 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.198788 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.198887 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.201016 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.201017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.201018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.201026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.202676 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.203193 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.203195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.203195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.203258 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.203786 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.203788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.203789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.203929 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.204015 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.204369 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.204460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 15:11:24.204727 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.194609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33379 (local address 127.21.204.66:45207)
0923 15:11:24.194758 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.194799 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.195279 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.195562 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.195567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.195870 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:24.196066 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.196071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.196960 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.196962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.198471 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.198475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.198910 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.198911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.199032 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.199548 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:24.199556 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:24.200913 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:24.202773 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.202780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.202782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.203095 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.203330 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.203332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.203332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.203473 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.204006 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:24.204013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:24.204202 (+ 189us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.204327 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.204492 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260923 15:11:24.205225 23779 heartbeater.cc:347] Connected to a master server at 127.21.204.126:33379
I20260923 15:11:24.205333 23779 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:24.205523 23779 heartbeater.cc:511] Master 127.21.204.126:33379 requested a full tablet report, sending...
I20260923 15:11:24.205981 23436 ts_manager.cc:194] Registered new tserver with Master: 26c3d1712d11414e9e263561ee53c9e1 (127.21.204.66:35981)
I20260923 15:11:24.206471 23436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:45207
I20260923 15:11:24.213236 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.206552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:57363 (local address 127.21.204.126:33379)
0923 15:11:24.206912 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.206914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.206922 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:24.206940 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.206940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.206951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.207063 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.207411 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.207893 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.209009 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.209195 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.209266 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.210382 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.210383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.210383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.210390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.211819 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.212232 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.212233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.212233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.212282 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.212517 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.212518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.212518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.212775 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.212847 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.213095 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.213168 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339}
I20260923 15:11:24.213286 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.206396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33379 (local address 127.21.204.66:57363)
0923 15:11:24.206546 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.206558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.206613 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.207078 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.207080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.207265 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:24.207328 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.207330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.207989 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.207990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.208873 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.208876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.209291 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.209293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.209377 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.209829 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:24.209835 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:24.210289 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:24.211907 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.211908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.211909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.212153 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.212351 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.212352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.212352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.212415 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.212864 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:24.212865 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:24.212915 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.213006 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.213140 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:24.233404 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33379
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:24.315915 23788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:24.316176 23788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:24.316218 23788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:24.317423 23788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:24.317497 23788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:24.317531 23788 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:24.317569 23788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:24.319080 23788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:33379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:24.319370 23788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:24.319628 23788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:24.325425 23794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:24.325456 23793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:24.325456 23796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:24.325740 23788 server_base.cc:1034] running on GCE node
I20260923 15:11:24.326068 23788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:24.326339 23788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:24.327545 23788 hybrid_clock.cc:648] HybridClock initialized: now 1790176284327533 us; error 31 us; skew 500 ppm
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:24.330307 23788 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:24.330823 23788 webserver.cc:460] Webserver started at http://127.21.204.67:42023/ using document root <none> and password file <none>
I20260923 15:11:24.331049 23788 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:24.331092 23788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:24.331182 23788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:24.332201 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "9f320d56bbf043baa5b186594384c286"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "dae9f9eb83ecf4a0d904b3369bc5b5c8"
server_key_iv: "3615aecbc064e8ee0b78fc53f922406d"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.332581 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "9f320d56bbf043baa5b186594384c286"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "dae9f9eb83ecf4a0d904b3369bc5b5c8"
server_key_iv: "3615aecbc064e8ee0b78fc53f922406d"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.334165 23788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:24.335124 23803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.335362 23788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:24.335546 23788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "9f320d56bbf043baa5b186594384c286"
format_stamp: "Formatted at 2026-09-23 15:11:24 on dist-test-slave-t5lt"
server_key: "dae9f9eb83ecf4a0d904b3369bc5b5c8"
server_key_iv: "3615aecbc064e8ee0b78fc53f922406d"
server_key_version: "encryptionkey@0"
I20260923 15:11:24.335670 23788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:24.343438 23788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:24.346109 23788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:24.346303 23788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:24.347039 23788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:24.347105 23788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.347155 23788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:24.347179 23788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.354204 23788 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:33123
I20260923 15:11:24.354521 23918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:33123 every 8 connection(s)
I20260923 15:11:24.354651 23788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:24.358403 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 23788
I20260923 15:11:24.358505 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:24.358673 22321 external_mini_cluster.cc:1428] Setting key f0c3d3c1a9c6de8af32e991cb1ef9fe2
I20260923 15:11:24.359072 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.347430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:44281 (local address 127.21.204.126:33379)
0923 15:11:24.347580 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.347583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.347835 (+ 252us) server_negotiation.cc:407] Connection header received
0923 15:11:24.348435 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.348437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.348451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.348537 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.349244 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.350123 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.351789 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.352172 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.352248 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.355563 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.355565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.355565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.355573 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.357225 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.357714 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.357715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.357716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.357774 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.358173 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.358175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.358175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.358388 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.358483 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.358881 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.358980 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260923 15:11:24.359274 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.347418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33379 (local address 127.21.204.67:44281)
0923 15:11:24.347685 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.347731 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.348329 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.348594 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.348598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.348910 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:24.349125 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.349129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.350231 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.350233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.351671 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.351675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.352344 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.352345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.352527 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.353275 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:24.353288 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:24.355376 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:24.357325 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.357332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.357334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.357571 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.357872 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.357874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.357874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.358075 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.358536 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:24.358544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:24.358702 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.358830 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.359002 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20260923 15:11:24.359766 23919 heartbeater.cc:347] Connected to a master server at 127.21.204.126:33379
I20260923 15:11:24.359849 23919 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:24.360042 23919 heartbeater.cc:511] Master 127.21.204.126:33379 requested a full tablet report, sending...
I20260923 15:11:24.360463 23436 ts_manager.cc:194] Registered new tserver with Master: 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123)
I20260923 15:11:24.360963 23436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:44281
I20260923 15:11:24.368603 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.360843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33379 (local address 127.21.204.67:39863)
0923 15:11:24.360989 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.361001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.361089 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.361277 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.361278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.361515 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:24.361581 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.361583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.362495 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.362496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.363005 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.363007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.363560 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.363562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.363698 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.364405 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:24.364413 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:24.364826 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:24.367189 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.367191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.367192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.367429 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.367806 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.367807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.367808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.367892 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.368273 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:24.368274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:24.368326 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.368401 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.368508 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260923 15:11:24.368896 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.360931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:39863 (local address 127.21.204.126:33379)
0923 15:11:24.360974 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.360976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.361060 (+ 84us) server_negotiation.cc:407] Connection header received
0923 15:11:24.361156 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.361158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.361169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.361226 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.361687 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.362353 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.363142 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.363343 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.363408 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.364996 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.364999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.364999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.365008 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.367063 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.367575 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.367577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.367578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.367669 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.368004 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.368006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.368006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.368146 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.368213 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.368640 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.368781 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:11:24.370731 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:24.396068 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.387877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.21.204.126:33379)
0923 15:11:24.387991 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.387993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.388018 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:11:24.388052 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.388053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.388064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.388123 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.388835 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.389520 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.390493 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.390738 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.390831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.393343 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.393345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.393345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.393353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.394916 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.395285 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.395285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.395286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.395339 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.395516 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.395517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.395517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.395626 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.395703 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.395885 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.395952 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 15:11:24.398232 23436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:24.399881 23436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:24.407181 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.402829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35981 (local address 127.0.0.1:40998)
0923 15:11:24.402979 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.403020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.403086 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.403595 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.403598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.403601 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:24.403697 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.403700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.404984 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.404985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.406116 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.406118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.406853 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.406947 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.406965 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.407032 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.407097 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":4,"thread_start_us":47,"threads_started":1}
I20260923 15:11:24.407258 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.403053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:33123 (local address 127.0.0.1:39738)
0923 15:11:24.403249 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.403258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.403299 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.403670 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.403671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.403673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:24.403729 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.403731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.404601 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.404602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.406055 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.406058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.406989 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.407049 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.407052 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.407123 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.407158 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":56,"threads_started":1}
I20260923 15:11:24.407284 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.403254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.21.204.67:33123)
0923 15:11:24.403413 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.403417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.403428 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:11:24.403506 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.403508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.403526 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.403600 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:24.403812 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.404509 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406145 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.406908 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406994 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.407048 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.407140 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.407183 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260923 15:11:24.407508 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.402922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.21.204.66:35981)
0923 15:11:24.403356 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.403362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.403373 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:11:24.403414 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.403416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.403434 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.403511 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:24.403781 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.404891 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406210 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.406754 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406840 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.406903 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.407387 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.407428 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":90,"threads_started":1}
I20260923 15:11:24.407897 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.402949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.21.204.65:43951)
0923 15:11:24.403424 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.403428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.403437 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:24.403474 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.403484 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.403500 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.403566 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:24.403781 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.404916 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406179 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.406908 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.406994 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.407048 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.407753 (+ 705us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.407786 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":95,"threads_started":1}
I20260923 15:11:24.408073 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.402846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:43951 (local address 127.0.0.1:40634)
0923 15:11:24.403165 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.403177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.403230 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.403632 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.403633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.403635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:24.403696 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.403699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.405003 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.405004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.406070 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.406073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.407080 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.407081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.407150 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.407154 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.407204 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.407988 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":24,"thread_start_us":101,"threads_started":1}
I20260923 15:11:24.408010 23853 tablet_service.cc:1467] Processing CreateTablet for tablet d385e8998aa642df9e645737cbd5267a (DEFAULT_TABLE table=test-table [id=4e1049d5afd541c9bc2156c616568f72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:24.408367 23853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d385e8998aa642df9e645737cbd5267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.411535 23713 tablet_service.cc:1467] Processing CreateTablet for tablet d385e8998aa642df9e645737cbd5267a (DEFAULT_TABLE table=test-table [id=4e1049d5afd541c9bc2156c616568f72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:24.411733 23573 tablet_service.cc:1467] Processing CreateTablet for tablet d385e8998aa642df9e645737cbd5267a (DEFAULT_TABLE table=test-table [id=4e1049d5afd541c9bc2156c616568f72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:24.411913 23713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d385e8998aa642df9e645737cbd5267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.412040 23573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d385e8998aa642df9e645737cbd5267a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.412675 23942 tablet_bootstrap.cc:492] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Bootstrap starting.
I20260923 15:11:24.413581 23942 tablet_bootstrap.cc:654] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.413976 23942 log.cc:824] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:24.414741 23942 tablet_bootstrap.cc:492] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: No bootstrap required, opened a new log
I20260923 15:11:24.414824 23942 ts_tablet_manager.cc:1397] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 15:11:24.415755 23945 tablet_bootstrap.cc:492] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Bootstrap starting.
I20260923 15:11:24.416424 23945 tablet_bootstrap.cc:654] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.416772 23945 log.cc:824] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:24.417107 23944 tablet_bootstrap.cc:492] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Bootstrap starting.
I20260923 15:11:24.417060 23942 raft_consensus.cc:348] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.417229 23942 raft_consensus.cc:374] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.417270 23942 raft_consensus.cc:729] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f320d56bbf043baa5b186594384c286, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.417395 23942 consensus_queue.cc:260] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.417660 23945 tablet_bootstrap.cc:492] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: No bootstrap required, opened a new log
I20260923 15:11:24.417665 23942 ts_tablet_manager.cc:1428] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 15:11:24.417732 23945 ts_tablet_manager.cc:1397] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:24.417748 23944 tablet_bootstrap.cc:654] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.417799 23919 heartbeater.cc:503] Master 127.21.204.126:33379 was elected leader, sending a full tablet report...
I20260923 15:11:24.418116 23944 log.cc:824] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:24.418957 23944 tablet_bootstrap.cc:492] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: No bootstrap required, opened a new log
I20260923 15:11:24.419024 23944 ts_tablet_manager.cc:1397] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:24.420269 23945 raft_consensus.cc:348] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.420425 23945 raft_consensus.cc:374] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.420449 23945 raft_consensus.cc:729] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f837811ea549359a0e45a97e180b57, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.420562 23945 consensus_queue.cc:260] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.420836 23945 ts_tablet_manager.cc:1428] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:24.420874 23639 heartbeater.cc:503] Master 127.21.204.126:33379 was elected leader, sending a full tablet report...
I20260923 15:11:24.421147 23944 raft_consensus.cc:348] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.421303 23944 raft_consensus.cc:374] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.421360 23944 raft_consensus.cc:729] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c3d1712d11414e9e263561ee53c9e1, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.421482 23944 consensus_queue.cc:260] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.421751 23944 ts_tablet_manager.cc:1428] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:24.421789 23779 heartbeater.cc:503] Master 127.21.204.126:33379 was elected leader, sending a full tablet report...
W20260923 15:11:24.452656 23780 tablet.cc:2369] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:24.546996 23640 tablet.cc:2369] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:24.605427 23920 tablet.cc:2369] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:24.686585 23950 raft_consensus.cc:482] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:24.686792 23950 raft_consensus.cc:504] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.687268 23950 leader_election.cc:288] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951), 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123)
I20260923 15:11:24.690455 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.687568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:46559 (local address 127.21.204.65:43951)
0923 15:11:24.687652 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.687654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.687664 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:24.687698 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.687699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.687710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.687768 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:24.687946 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.688605 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.689604 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.690079 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.690170 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.690179 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.690349 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.690389 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 15:11:24.690455 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.687529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:43951 (local address 127.21.204.66:46559)
0923 15:11:24.687573 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.687586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.687634 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.687783 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.687784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.687786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:24.687856 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.687857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.688726 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.688728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.689505 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.689507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.690169 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.690170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.690259 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.690262 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.690339 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.690381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 15:11:24.690559 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.687790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:47331 (local address 127.21.204.67:33123)
0923 15:11:24.687846 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.687848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.687857 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:24.687906 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.687907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.687917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.687969 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:24.688136 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.688764 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.689756 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.690197 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.690288 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.690297 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.690458 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.690493 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 15:11:24.690577 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.687650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:33123 (local address 127.21.204.66:47331)
0923 15:11:24.687785 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.687796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.687845 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.687981 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.687982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.687984 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:24.688060 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.688062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.688878 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.688880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.689655 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.689658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.690288 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.690289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.690368 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.690372 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.690451 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.690491 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260923 15:11:24.690842 23593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d385e8998aa642df9e645737cbd5267a" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f837811ea549359a0e45a97e180b57" is_pre_election: true
I20260923 15:11:24.690909 23873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d385e8998aa642df9e645737cbd5267a" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f320d56bbf043baa5b186594384c286" is_pre_election: true
I20260923 15:11:24.691061 23873 raft_consensus.cc:2454] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 0.
I20260923 15:11:24.691061 23593 raft_consensus.cc:2454] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 0.
I20260923 15:11:24.691305 23664 leader_election.cc:302] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c3d1712d11414e9e263561ee53c9e1, 44f837811ea549359a0e45a97e180b57; no voters:
I20260923 15:11:24.691466 23950 raft_consensus.cc:2781] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:24.691515 23950 raft_consensus.cc:482] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:24.691541 23950 raft_consensus.cc:3037] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:24.692546 23950 raft_consensus.cc:504] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.692703 23950 leader_election.cc:288] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951), 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123)
I20260923 15:11:24.692869 23593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d385e8998aa642df9e645737cbd5267a" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f837811ea549359a0e45a97e180b57"
I20260923 15:11:24.692927 23873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d385e8998aa642df9e645737cbd5267a" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f320d56bbf043baa5b186594384c286"
I20260923 15:11:24.692988 23593 raft_consensus.cc:3037] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:24.693027 23873 raft_consensus.cc:3037] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:24.693727 23873 raft_consensus.cc:2454] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 1.
I20260923 15:11:24.693730 23593 raft_consensus.cc:2454] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 1.
I20260923 15:11:24.693938 23664 leader_election.cc:302] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c3d1712d11414e9e263561ee53c9e1, 44f837811ea549359a0e45a97e180b57; no voters:
I20260923 15:11:24.694069 23950 raft_consensus.cc:2781] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:24.694240 23950 raft_consensus.cc:686] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 26c3d1712d11414e9e263561ee53c9e1, State: Running, Role: LEADER
I20260923 15:11:24.694352 23950 consensus_queue.cc:237] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.695478 23436 catalog_manager.cc:5495] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c3d1712d11414e9e263561ee53c9e1 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:24.763839 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.760576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.21.204.66:35981)
0923 15:11:24.760661 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.760663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.760675 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:11:24.760714 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.760716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.760740 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.760808 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:24.761013 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.761737 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.762916 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.763118 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.763179 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.763428 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:24.763573 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:24.763684 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.763713 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.763739 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 15:11:24.765719 23873 raft_consensus.cc:1261] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Refusing update from remote peer 26c3d1712d11414e9e263561ee53c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:24.766134 23593 raft_consensus.cc:1261] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Refusing update from remote peer 26c3d1712d11414e9e263561ee53c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:24.766109 23952 consensus_queue.cc:1035] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:24.766403 23950 consensus_queue.cc:1035] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:24.770603 23956 mvcc.cc:205] Tried to move back new op lower bound from 7332562062397198336 to 7332562062108782592. Current Snapshot: MvccSnapshot[applied={T|T < 7332562062397198336 or (T in {7332562062397198336})}]
I20260923 15:11:24.773617 23435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:24.773716 23435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:24.774673 23435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3: Sending DeleteTablet for 3 replicas of tablet d385e8998aa642df9e645737cbd5267a
I20260923 15:11:24.775050 23713 tablet_service.cc:1514] Processing DeleteTablet for tablet d385e8998aa642df9e645737cbd5267a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:24 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:40998
I20260923 15:11:24.775245 23853 tablet_service.cc:1514] Processing DeleteTablet for tablet d385e8998aa642df9e645737cbd5267a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:24 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:39738
I20260923 15:11:24.775247 23573 tablet_service.cc:1514] Processing DeleteTablet for tablet d385e8998aa642df9e645737cbd5267a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:24 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:40634
I20260923 15:11:24.775293 23965 tablet_replica.cc:330] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: stopping tablet replica
I20260923 15:11:24.775388 23965 raft_consensus.cc:2229] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:24.775485 23965 raft_consensus.cc:2258] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:24.775940 23435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 15:11:24.776031 23965 ts_tablet_manager.cc:1905] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 15:11:24.776113 23435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:24.777318 23966 tablet_replica.cc:330] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: stopping tablet replica
I20260923 15:11:24.777500 23966 raft_consensus.cc:2229] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:24.777644 23573 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0c272fd5d04471870d59afbdce0ae6 (DEFAULT_TABLE table=test-table [id=96fda800767e499bac1af1351cc952ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:24.777760 23573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0c272fd5d04471870d59afbdce0ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.777750 23966 raft_consensus.cc:2258] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:24.778182 23853 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0c272fd5d04471870d59afbdce0ae6 (DEFAULT_TABLE table=test-table [id=96fda800767e499bac1af1351cc952ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:24.778200 23965 ts_tablet_manager.cc:1918] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:24.778303 23853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0c272fd5d04471870d59afbdce0ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.778379 23965 log.cc:1196] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/d385e8998aa642df9e645737cbd5267a
I20260923 15:11:24.779027 23965 ts_tablet_manager.cc:1939] T d385e8998aa642df9e645737cbd5267a P 26c3d1712d11414e9e263561ee53c9e1: Deleting consensus metadata
I20260923 15:11:24.779482 23967 tablet_replica.cc:330] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: stopping tablet replica
I20260923 15:11:24.779656 23967 raft_consensus.cc:2229] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:24.779814 23942 tablet_bootstrap.cc:492] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: Bootstrap starting.
I20260923 15:11:24.779875 23945 tablet_bootstrap.cc:492] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: Bootstrap starting.
I20260923 15:11:24.779836 23967 raft_consensus.cc:2258] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:24.780284 23945 tablet_bootstrap.cc:654] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.780288 23942 tablet_bootstrap.cc:654] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.780843 23421 catalog_manager.cc:4841] TS 26c3d1712d11414e9e263561ee53c9e1 (127.21.204.66:35981): tablet d385e8998aa642df9e645737cbd5267a (table test-table [id=4e1049d5afd541c9bc2156c616568f72]) successfully deleted
I20260923 15:11:24.781013 23713 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0c272fd5d04471870d59afbdce0ae6 (DEFAULT_TABLE table=test-table [id=96fda800767e499bac1af1351cc952ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:24.781129 23713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0c272fd5d04471870d59afbdce0ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.781267 23967 ts_tablet_manager.cc:1905] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:24.782737 23944 tablet_bootstrap.cc:492] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: Bootstrap starting.
I20260923 15:11:24.783169 23944 tablet_bootstrap.cc:654] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.783346 23945 tablet_bootstrap.cc:492] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: No bootstrap required, opened a new log
I20260923 15:11:24.783407 23945 ts_tablet_manager.cc:1397] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 15:11:24.783546 23967 ts_tablet_manager.cc:1918] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:24.783588 23945 raft_consensus.cc:348] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.783670 23945 raft_consensus.cc:374] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.783699 23945 raft_consensus.cc:729] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f837811ea549359a0e45a97e180b57, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.783741 23945 consensus_queue.cc:260] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.783787 23967 log.cc:1196] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/d385e8998aa642df9e645737cbd5267a
I20260923 15:11:24.783883 23945 ts_tablet_manager.cc:1428] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:24.783963 23944 tablet_bootstrap.cc:492] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: No bootstrap required, opened a new log
I20260923 15:11:24.784018 23944 ts_tablet_manager.cc:1397] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:24.784168 23944 raft_consensus.cc:348] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.784242 23944 raft_consensus.cc:374] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.784269 23944 raft_consensus.cc:729] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c3d1712d11414e9e263561ee53c9e1, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.784308 23944 consensus_queue.cc:260] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.784392 23944 ts_tablet_manager.cc:1428] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:24.784473 23967 ts_tablet_manager.cc:1939] T d385e8998aa642df9e645737cbd5267a P 9f320d56bbf043baa5b186594384c286: Deleting consensus metadata
I20260923 15:11:24.784720 23966 ts_tablet_manager.cc:1905] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:24.784999 23420 catalog_manager.cc:4841] TS 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123): tablet d385e8998aa642df9e645737cbd5267a (table test-table [id=4e1049d5afd541c9bc2156c616568f72]) successfully deleted
I20260923 15:11:24.786540 23966 ts_tablet_manager.cc:1918] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:24.786634 23966 log.cc:1196] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/d385e8998aa642df9e645737cbd5267a
I20260923 15:11:24.787345 23966 ts_tablet_manager.cc:1939] T d385e8998aa642df9e645737cbd5267a P 44f837811ea549359a0e45a97e180b57: Deleting consensus metadata
I20260923 15:11:24.787840 23420 catalog_manager.cc:4841] TS 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951): tablet d385e8998aa642df9e645737cbd5267a (table test-table [id=4e1049d5afd541c9bc2156c616568f72]) successfully deleted
I20260923 15:11:24.790262 23942 tablet_bootstrap.cc:492] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: No bootstrap required, opened a new log
I20260923 15:11:24.790314 23942 ts_tablet_manager.cc:1397] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260923 15:11:24.790473 23942 raft_consensus.cc:348] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.790544 23942 raft_consensus.cc:374] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.790562 23942 raft_consensus.cc:729] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f320d56bbf043baa5b186594384c286, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.790609 23942 consensus_queue.cc:260] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.790699 23942 ts_tablet_manager.cc:1428] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:24.900168 23950 raft_consensus.cc:482] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:24.900274 23950 raft_consensus.cc:504] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.900543 23950 leader_election.cc:288] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951), 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123)
I20260923 15:11:24.900810 23593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c272fd5d04471870d59afbdce0ae6" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f837811ea549359a0e45a97e180b57" is_pre_election: true
I20260923 15:11:24.900808 23873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c272fd5d04471870d59afbdce0ae6" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f320d56bbf043baa5b186594384c286" is_pre_election: true
I20260923 15:11:24.900918 23873 raft_consensus.cc:2454] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 0.
I20260923 15:11:24.900918 23593 raft_consensus.cc:2454] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 0.
I20260923 15:11:24.901090 23664 leader_election.cc:302] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c3d1712d11414e9e263561ee53c9e1, 9f320d56bbf043baa5b186594384c286; no voters:
I20260923 15:11:24.901224 23950 raft_consensus.cc:2781] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:24.901284 23950 raft_consensus.cc:482] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:24.901315 23950 raft_consensus.cc:3037] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:24.902045 23950 raft_consensus.cc:504] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.902192 23950 leader_election.cc:288] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951), 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123)
I20260923 15:11:24.902469 23873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c272fd5d04471870d59afbdce0ae6" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f320d56bbf043baa5b186594384c286"
I20260923 15:11:24.902554 23873 raft_consensus.cc:3037] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:24.902464 23593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0c272fd5d04471870d59afbdce0ae6" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f837811ea549359a0e45a97e180b57"
I20260923 15:11:24.902590 23593 raft_consensus.cc:3037] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:24.903311 23873 raft_consensus.cc:2454] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 1.
I20260923 15:11:24.903311 23593 raft_consensus.cc:2454] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 1.
I20260923 15:11:24.903522 23664 leader_election.cc:302] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c3d1712d11414e9e263561ee53c9e1, 44f837811ea549359a0e45a97e180b57; no voters:
I20260923 15:11:24.903678 23950 raft_consensus.cc:2781] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:24.903761 23950 raft_consensus.cc:686] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 26c3d1712d11414e9e263561ee53c9e1, State: Running, Role: LEADER
I20260923 15:11:24.903818 23950 consensus_queue.cc:237] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.904412 23435 catalog_manager.cc:5495] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c3d1712d11414e9e263561ee53c9e1 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } health_report { overall_health: UNKNOWN } } }
Sep 23 15:11:24 dist-test-slave-t5lt krb5kdc[23391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176283, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:24.978137 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.970764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33379 (local address 127.0.0.1:39376)
0923 15:11:24.970917 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:24.970931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:24.970990 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:24.971398 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:24.971399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:24.971732 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:24.971779 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.971781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.972454 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.972455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.972972 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:24.972974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.973335 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.973335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:24.973381 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.973938 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:24.973943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:24.975322 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:24.976956 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.976966 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.976968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.977188 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.977415 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:24.977415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:24.977416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:24.977476 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:24.977724 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:24.977733 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:24.977779 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 15:11:24.977864 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:11:24.978009 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260923 15:11:24.978157 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:24.970976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.21.204.126:33379)
0923 15:11:24.971176 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:24.971178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:24.971184 (+ 6us) server_negotiation.cc:407] Connection header received
0923 15:11:24.971204 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:24.971205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:24.971213 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:24.971393 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:24.971888 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.972376 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.973059 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:24.973248 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:24.973334 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:24.975413 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:24.975415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:24.975415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:24.975422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:24.976854 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.977283 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.977284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.977284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.977347 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:24.977539 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:24.977540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:24.977541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:24.977657 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:24.977735 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:24.977931 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 15:11:24.978062 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260923 15:11:24.980324 23435 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:39376:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 15:11:24.980574 23435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:24.982260 23853 tablet_service.cc:1467] Processing CreateTablet for tablet 54922461492843b2a72acbc6b6184314 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=977f139c5cc2416aaaa17dc82d5a49ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:24.982260 23573 tablet_service.cc:1467] Processing CreateTablet for tablet 54922461492843b2a72acbc6b6184314 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=977f139c5cc2416aaaa17dc82d5a49ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:24.982261 23713 tablet_service.cc:1467] Processing CreateTablet for tablet 54922461492843b2a72acbc6b6184314 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=977f139c5cc2416aaaa17dc82d5a49ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:24.982430 23853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54922461492843b2a72acbc6b6184314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.982430 23573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54922461492843b2a72acbc6b6184314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.982432 23713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54922461492843b2a72acbc6b6184314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:24.983922 23944 tablet_bootstrap.cc:492] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: Bootstrap starting.
I20260923 15:11:24.983999 23942 tablet_bootstrap.cc:492] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: Bootstrap starting.
I20260923 15:11:24.984323 23944 tablet_bootstrap.cc:654] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.984380 23942 tablet_bootstrap.cc:654] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.984717 23945 tablet_bootstrap.cc:492] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: Bootstrap starting.
I20260923 15:11:24.985081 23942 tablet_bootstrap.cc:492] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: No bootstrap required, opened a new log
I20260923 15:11:24.985108 23944 tablet_bootstrap.cc:492] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: No bootstrap required, opened a new log
I20260923 15:11:24.985154 23944 ts_tablet_manager.cc:1397] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:24.985155 23942 ts_tablet_manager.cc:1397] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:24.985179 23945 tablet_bootstrap.cc:654] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:24.985288 23944 raft_consensus.cc:348] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.985392 23942 raft_consensus.cc:348] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.985442 23944 raft_consensus.cc:374] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.985478 23942 raft_consensus.cc:374] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.985509 23942 raft_consensus.cc:729] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f320d56bbf043baa5b186594384c286, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.985484 23944 raft_consensus.cc:729] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c3d1712d11414e9e263561ee53c9e1, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.985553 23942 consensus_queue.cc:260] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.985647 23942 ts_tablet_manager.cc:1428] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 15:11:24.985639 23944 consensus_queue.cc:260] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.985752 23944 ts_tablet_manager.cc:1428] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:24.985901 23947 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:24.985967 23950 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:24.986020 23947 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:24.986042 23950 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:24.986562 23945 tablet_bootstrap.cc:492] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: No bootstrap required, opened a new log
I20260923 15:11:24.986613 23945 ts_tablet_manager.cc:1397] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:24.986755 23945 raft_consensus.cc:348] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.986850 23945 raft_consensus.cc:374] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:24.986881 23945 raft_consensus.cc:729] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f837811ea549359a0e45a97e180b57, State: Initialized, Role: FOLLOWER
I20260923 15:11:24.986925 23945 consensus_queue.cc:260] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:24.987026 23945 ts_tablet_manager.cc:1428] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:24.987109 23949 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:24.987224 23949 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:25.302178 23950 consensus_queue.cc:1035] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:25.304087 23950 consensus_queue.cc:1035] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:25.352365 23952 raft_consensus.cc:482] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:25.352479 23952 raft_consensus.cc:504] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:25.352676 23952 leader_election.cc:288] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951), 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123)
I20260923 15:11:25.352945 23593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54922461492843b2a72acbc6b6184314" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f837811ea549359a0e45a97e180b57" is_pre_election: true
I20260923 15:11:25.353072 23593 raft_consensus.cc:2454] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 0.
I20260923 15:11:25.353013 23873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54922461492843b2a72acbc6b6184314" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f320d56bbf043baa5b186594384c286" is_pre_election: true
I20260923 15:11:25.353155 23873 raft_consensus.cc:2454] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 0.
I20260923 15:11:25.353247 23664 leader_election.cc:302] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c3d1712d11414e9e263561ee53c9e1, 44f837811ea549359a0e45a97e180b57; no voters:
I20260923 15:11:25.353368 23952 raft_consensus.cc:2781] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:25.353436 23952 raft_consensus.cc:482] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:25.353466 23952 raft_consensus.cc:3037] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:25.354315 23952 raft_consensus.cc:504] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:25.354463 23952 leader_election.cc:288] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951), 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123)
I20260923 15:11:25.354657 23873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54922461492843b2a72acbc6b6184314" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f320d56bbf043baa5b186594384c286"
I20260923 15:11:25.354653 23593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54922461492843b2a72acbc6b6184314" candidate_uuid: "26c3d1712d11414e9e263561ee53c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44f837811ea549359a0e45a97e180b57"
I20260923 15:11:25.354743 23593 raft_consensus.cc:3037] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:25.354758 23873 raft_consensus.cc:3037] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:25.355362 23873 raft_consensus.cc:2454] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 1.
I20260923 15:11:25.355509 23593 raft_consensus.cc:2454] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26c3d1712d11414e9e263561ee53c9e1 in term 1.
I20260923 15:11:25.355566 23664 leader_election.cc:302] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c3d1712d11414e9e263561ee53c9e1, 9f320d56bbf043baa5b186594384c286; no voters:
I20260923 15:11:25.355648 23952 raft_consensus.cc:2781] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:25.355701 23952 raft_consensus.cc:686] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 26c3d1712d11414e9e263561ee53c9e1, State: Running, Role: LEADER
I20260923 15:11:25.355755 23952 consensus_queue.cc:237] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } }
I20260923 15:11:25.356076 23950 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: TxnStatusTablet state changed. Reason: New leader 26c3d1712d11414e9e263561ee53c9e1. Latest consensus state: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:25.356154 23950 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:25.356391 23988 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 15:11:25.356503 23988 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 15:11:25.356436 23436 catalog_manager.cc:5495] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26c3d1712d11414e9e263561ee53c9e1 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:25.428785 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:25.426185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35981 (local address 127.0.0.1:41008)
0923 15:11:25.426270 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:25.426284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:25.426332 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:25.426634 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:25.426635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:25.426637 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:25.426689 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:25.426692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.427357 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.427364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:25.427949 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:25.427950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.428329 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.428330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:25.428387 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:25.428411 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:25.428609 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:25.428621 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 15:11:25.428667 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 15:11:25.428719 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 15:11:25.428782 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:25.426291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.21.204.66:35981)
0923 15:11:25.426468 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:25.426470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:25.426477 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:25.426510 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:25.426511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:25.426522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:25.426616 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:25.426776 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.427268 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.428044 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.428252 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.428330 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:25.428497 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:25.428543 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:25.428606 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:25.428669 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 15:11:25.428692 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260923 15:11:25.429986 23873 raft_consensus.cc:1261] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Refusing update from remote peer 26c3d1712d11414e9e263561ee53c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:25.429992 23593 raft_consensus.cc:1261] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Refusing update from remote peer 26c3d1712d11414e9e263561ee53c9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:25.430238 23950 consensus_queue.cc:1035] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:25.430339 23952 consensus_queue.cc:1035] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:25.430932 23947 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: TxnStatusTablet state changed. Reason: New leader 26c3d1712d11414e9e263561ee53c9e1. Latest consensus state: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:25.431044 23947 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:25.431121 23949 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: TxnStatusTablet state changed. Reason: New leader 26c3d1712d11414e9e263561ee53c9e1. Latest consensus state: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:25.431210 23949 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:25.432049 23952 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:25.432134 23952 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:25.432192 23983 mvcc.cc:205] Tried to move back new op lower bound from 7332562065119191040 to 7332562064817680384. Current Snapshot: MvccSnapshot[applied={T|T < 7332562065119191040 or (T in {7332562065119191040})}]
I20260923 15:11:25.432170 23949 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:25.432257 23949 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 44f837811ea549359a0e45a97e180b57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:25.432276 23986 mvcc.cc:205] Tried to move back new op lower bound from 7332562065119191040 to 7332562064817680384. Current Snapshot: MvccSnapshot[applied={T|T < 7332562065119191040}]
I20260923 15:11:25.432646 23950 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:25.432713 23950 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 26c3d1712d11414e9e263561ee53c9e1: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:25.433463 23947 tablet_replica.cc:439] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26c3d1712d11414e9e263561ee53c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f837811ea549359a0e45a97e180b57" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43951 } } peers { permanent_uuid: "26c3d1712d11414e9e263561ee53c9e1" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35981 } } peers { permanent_uuid: "9f320d56bbf043baa5b186594384c286" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33123 } } }
I20260923 15:11:25.433540 23947 tablet_replica.cc:442] T 54922461492843b2a72acbc6b6184314 P 9f320d56bbf043baa5b186594384c286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:25.439433 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:25.436535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:45549 (local address 127.21.204.66:35981)
0923 15:11:25.436574 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:25.436576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:25.436605 (+ 29us) server_negotiation.cc:407] Connection header received
0923 15:11:25.436675 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:25.436676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:25.436686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:25.436743 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:25.436853 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.437573 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.438531 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.438824 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.438904 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:25.439088 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:25.439121 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:25.439180 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:25.439301 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 15:11:25.439327 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260923 15:11:25.439435 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:25.436382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35981 (local address 127.21.204.66:45549)
0923 15:11:25.436520 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:25.436530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:25.436615 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:25.436745 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:25.436746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:25.436747 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:25.436814 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:25.436815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.437721 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.437722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:25.438389 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:25.438391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.438908 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.438909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:25.438992 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:25.439009 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:25.439212 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:25.439221 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 15:11:25.439280 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:11:25.439327 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":14,"thread_start_us":84,"threads_started":1}
I20260923 15:11:25.455235 23435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:25.455364 23435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:25.456099 23435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d25d2d852e004a0298931e88579ca6a3: Sending DeleteTablet for 3 replicas of tablet 9a0c272fd5d04471870d59afbdce0ae6
I20260923 15:11:25.456399 23853 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0c272fd5d04471870d59afbdce0ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:25 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:39738
I20260923 15:11:25.456410 23573 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0c272fd5d04471870d59afbdce0ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:25 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:40634
I20260923 15:11:25.456619 23712 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0c272fd5d04471870d59afbdce0ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:25 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:40998
I20260923 15:11:25.456666 24003 tablet_replica.cc:330] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: stopping tablet replica
I20260923 15:11:25.456730 24003 raft_consensus.cc:2229] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:25.456785 24003 raft_consensus.cc:2258] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:25.456941 24004 tablet_replica.cc:330] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: stopping tablet replica
I20260923 15:11:25.457023 24004 raft_consensus.cc:2229] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:25.457122 24004 raft_consensus.cc:2258] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:25.457262 24003 ts_tablet_manager.cc:1905] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:25.457381 24004 ts_tablet_manager.cc:1905] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:25.457501 24002 tablet_replica.cc:330] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: stopping tablet replica
I20260923 15:11:25.457576 24002 raft_consensus.cc:2229] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:25.457626 24002 raft_consensus.cc:2258] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:25.457899 24002 ts_tablet_manager.cc:1905] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:25.459367 24004 ts_tablet_manager.cc:1918] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:25.459432 24004 log.cc:1196] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/9a0c272fd5d04471870d59afbdce0ae6
I20260923 15:11:25.459693 24004 ts_tablet_manager.cc:1939] T 9a0c272fd5d04471870d59afbdce0ae6 P 26c3d1712d11414e9e263561ee53c9e1: Deleting consensus metadata
I20260923 15:11:25.460034 24003 ts_tablet_manager.cc:1918] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:25.460216 23421 catalog_manager.cc:4841] TS 26c3d1712d11414e9e263561ee53c9e1 (127.21.204.66:35981): tablet 9a0c272fd5d04471870d59afbdce0ae6 (table test-table [id=96fda800767e499bac1af1351cc952ef]) successfully deleted
I20260923 15:11:25.460270 24003 log.cc:1196] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/9a0c272fd5d04471870d59afbdce0ae6
I20260923 15:11:25.460450 24002 ts_tablet_manager.cc:1918] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:25.460513 24002 log.cc:1196] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/9a0c272fd5d04471870d59afbdce0ae6
I20260923 15:11:25.460750 24003 ts_tablet_manager.cc:1939] T 9a0c272fd5d04471870d59afbdce0ae6 P 9f320d56bbf043baa5b186594384c286: Deleting consensus metadata
I20260923 15:11:25.460750 24002 ts_tablet_manager.cc:1939] T 9a0c272fd5d04471870d59afbdce0ae6 P 44f837811ea549359a0e45a97e180b57: Deleting consensus metadata
I20260923 15:11:25.461201 23420 catalog_manager.cc:4841] TS 9f320d56bbf043baa5b186594384c286 (127.21.204.67:33123): tablet 9a0c272fd5d04471870d59afbdce0ae6 (table test-table [id=96fda800767e499bac1af1351cc952ef]) successfully deleted
I20260923 15:11:25.461290 23420 catalog_manager.cc:4841] TS 44f837811ea549359a0e45a97e180b57 (127.21.204.65:43951): tablet 9a0c272fd5d04471870d59afbdce0ae6 (table test-table [id=96fda800767e499bac1af1351cc952ef]) successfully deleted
Sep 23 15:11:25 dist-test-slave-t5lt krb5kdc[23391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176284, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:25.470458 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:25.461281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.21.204.65:43951)
0923 15:11:25.461979 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:25.461981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:25.461990 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:25.462013 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:25.462013 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:25.462048 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:25.462121 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:25.462697 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.463426 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.464154 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.464448 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.464528 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:25.467131 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:25.467144 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:25.467145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:25.467153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:25.469148 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:25.469528 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:25.469532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:25.469532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:25.469626 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:25.469855 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:25.469856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:25.469857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:25.470063 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:25.470198 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:25.470266 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 15:11:25.470344 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":58,"threads_started":1}
W20260923 15:11:25.470736 23633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40638
I20260923 15:11:25.478600 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:25.471752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.21.204.126:33379)
0923 15:11:25.471801 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:25.471804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:25.471860 (+ 56us) server_negotiation.cc:407] Connection header received
0923 15:11:25.471928 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:25.471929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:25.471942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:25.471997 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:25.472480 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.473243 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.474026 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:25.474345 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:25.474420 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:25.475757 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:25.475759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:25.475759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:25.475766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:25.477296 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:25.477664 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:25.477665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:25.477666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:25.477727 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:25.477992 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:25.477993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:25.477993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:25.478135 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:25.478237 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:25.478401 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 15:11:25.478494 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260923 15:11:25.478859 23435 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39386
I20260923 15:11:25.479427 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 23508
I20260923 15:11:25.486691 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 23648
I20260923 15:11:25.492508 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 23788
I20260923 15:11:25.496867 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 23404
2026-09-23T15:11:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2934 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:25 dist-test-slave-t5lt krb5kdc[24012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:25 dist-test-slave-t5lt krb5kdc[24012](info): set up 2 sockets
Sep 23 15:11:25 dist-test-slave-t5lt krb5kdc[24012](info): commencing operation
krb5kdc: starting...
W20260923 15:11:26.550920 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:26 dist-test-slave-t5lt krb5kdc[24012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:26.665213 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:37143
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:26.745998 24025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:26.746214 24025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:26.746236 24025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:26.747373 24025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:26.747414 24025 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 15:11:26.747428 24025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:26.747442 24025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:26.747453 24025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:26.747465 24025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:26.749137 24025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:37143
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:26.749428 24025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:26.749804 24025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:26.754930 24033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:26.754926 24031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:26.754956 24025 server_base.cc:1034] running on GCE node
W20260923 15:11:26.754926 24030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:26.755441 24025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:26.755796 24025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:26.756963 24025 hybrid_clock.cc:648] HybridClock initialized: now 1790176286756951 us; error 32 us; skew 500 ppm
Sep 23 15:11:26 dist-test-slave-t5lt krb5kdc[24012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176286, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:26.759469 24025 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:26.759847 24025 webserver.cc:460] Webserver started at http://127.21.204.126:41417/ using document root <none> and password file <none>
I20260923 15:11:26.760088 24025 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:26.760133 24025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:26.760265 24025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:26.761327 24025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "1cccebd1fd124d85993a808f5bfe141f"
format_stamp: "Formatted at 2026-09-23 15:11:26 on dist-test-slave-t5lt"
server_key: "3aa74511a44b70be70055d433a3d316f"
server_key_iv: "f32c2a78d33019818cc7431e68938368"
server_key_version: "encryptionkey@0"
I20260923 15:11:26.761693 24025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "1cccebd1fd124d85993a808f5bfe141f"
format_stamp: "Formatted at 2026-09-23 15:11:26 on dist-test-slave-t5lt"
server_key: "3aa74511a44b70be70055d433a3d316f"
server_key_iv: "f32c2a78d33019818cc7431e68938368"
server_key_version: "encryptionkey@0"
I20260923 15:11:26.763065 24025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:26.763931 24040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:26.764209 24025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:26.764292 24025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "1cccebd1fd124d85993a808f5bfe141f"
format_stamp: "Formatted at 2026-09-23 15:11:26 on dist-test-slave-t5lt"
server_key: "3aa74511a44b70be70055d433a3d316f"
server_key_iv: "f32c2a78d33019818cc7431e68938368"
server_key_version: "encryptionkey@0"
I20260923 15:11:26.764385 24025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:26.791313 24025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:26.793653 24025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:26.793771 24025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:26.798609 24025 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:37143
I20260923 15:11:26.798691 24102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:37143 every 8 connection(s)
I20260923 15:11:26.799086 24025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:26.799901 24103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:26.800232 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 24025
I20260923 15:11:26.800331 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:26.800509 22321 external_mini_cluster.cc:1428] Setting key 108d6f3b8e615a945a2f776910171b45
I20260923 15:11:26.803290 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Bootstrap starting.
I20260923 15:11:26.803947 24103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:26.804275 24103 log.cc:824] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:26.805256 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: No bootstrap required, opened a new log
I20260923 15:11:26.807590 24103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cccebd1fd124d85993a808f5bfe141f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 37143 } }
I20260923 15:11:26.807768 24103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:26.807812 24103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cccebd1fd124d85993a808f5bfe141f, State: Initialized, Role: FOLLOWER
I20260923 15:11:26.807948 24103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cccebd1fd124d85993a808f5bfe141f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 37143 } }
I20260923 15:11:26.808045 24103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 23 15:11:26 dist-test-slave-t5lt krb5kdc[24012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:26.808077 24103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:26.808121 24103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:26.808821 24103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cccebd1fd124d85993a808f5bfe141f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 37143 } }
I20260923 15:11:26.808912 24103 leader_election.cc:302] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cccebd1fd124d85993a808f5bfe141f; no voters:
I20260923 15:11:26.809116 24103 leader_election.cc:288] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:26.809232 24108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:26.809381 24108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [term 1 LEADER]: Becoming Leader. State: Replica: 1cccebd1fd124d85993a808f5bfe141f, State: Running, Role: LEADER
I20260923 15:11:26.809451 24103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:26.809478 24108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cccebd1fd124d85993a808f5bfe141f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 37143 } }
I20260923 15:11:26.811021 24110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cccebd1fd124d85993a808f5bfe141f. Latest consensus state: current_term: 1 leader_uuid: "1cccebd1fd124d85993a808f5bfe141f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cccebd1fd124d85993a808f5bfe141f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 37143 } } }
I20260923 15:11:26.811141 24110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:26.811720 24109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cccebd1fd124d85993a808f5bfe141f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cccebd1fd124d85993a808f5bfe141f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 37143 } } }
I20260923 15:11:26.811815 24109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:26.811929 24123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:26.812414 24123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:26.812704 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:26.801159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.21.204.126:37143)
0923 15:11:26.801347 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:26.801356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:26.801392 (+ 36us) server_negotiation.cc:407] Connection header received
0923 15:11:26.801805 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:26.801811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:26.801857 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:26.802045 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:26.802869 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:26.803842 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:26.804905 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:26.805624 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:26.805749 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:26.808683 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:26.808687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:26.808687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:26.808698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:26.810839 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:26.811273 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:26.811277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:26.811277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:26.811394 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:26.811737 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:26.811738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:26.811739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:26.812062 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:26.812227 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:26.812359 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 15:11:26.812504 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260923 15:11:26.813961 24123 catalog_manager.cc:1347] Generated new cluster ID: 32584aa36ccf455ca807455aa5c2ec9a
I20260923 15:11:26.814014 24123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:26.829329 24123 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:26.830000 24123 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:26.835811 24123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Generated new TSK 0
I20260923 15:11:26.836011 24123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:26.872464 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:37143
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:26.976572 24129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:26.976799 24129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:26.976826 24129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:26.978145 24129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:26.978185 24129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:26.978207 24129 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 15:11:26.978219 24129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:26.978236 24129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:26.979725 24129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:26.979971 24129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:26.980217 24129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:11:26.985366 24129 server_base.cc:1034] running on GCE node
W20260923 15:11:26.985375 24135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:26.985378 24134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:26.985553 24137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:26.985798 24129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:26.986076 24129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:26.987242 24129 hybrid_clock.cc:648] HybridClock initialized: now 1790176286987215 us; error 37 us; skew 500 ppm
Sep 23 15:11:26 dist-test-slave-t5lt krb5kdc[24012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176286, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:26.990175 24129 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:26.990583 24129 webserver.cc:460] Webserver started at http://127.21.204.65:40583/ using document root <none> and password file <none>
I20260923 15:11:26.990839 24129 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:26.990887 24129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:26.991056 24129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:26.992179 24129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "7974ed8621194b3fa1128f5e200db1ac"
format_stamp: "Formatted at 2026-09-23 15:11:26 on dist-test-slave-t5lt"
server_key: "0d2a87da62830ab3c7d31f29689e6bd1"
server_key_iv: "48080d40416f79604729a3b09b03a523"
server_key_version: "encryptionkey@0"
I20260923 15:11:26.992565 24129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "7974ed8621194b3fa1128f5e200db1ac"
format_stamp: "Formatted at 2026-09-23 15:11:26 on dist-test-slave-t5lt"
server_key: "0d2a87da62830ab3c7d31f29689e6bd1"
server_key_iv: "48080d40416f79604729a3b09b03a523"
server_key_version: "encryptionkey@0"
I20260923 15:11:26.994155 24129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:26.994979 24144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:26.995235 24129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:26.995316 24129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "7974ed8621194b3fa1128f5e200db1ac"
format_stamp: "Formatted at 2026-09-23 15:11:26 on dist-test-slave-t5lt"
server_key: "0d2a87da62830ab3c7d31f29689e6bd1"
server_key_iv: "48080d40416f79604729a3b09b03a523"
server_key_version: "encryptionkey@0"
I20260923 15:11:26.995415 24129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:27.017161 24129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:27.019660 24129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:27.019810 24129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:27.020360 24129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:27.020397 24129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.020419 24129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:27.020475 24129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.027662 24129 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:44621
I20260923 15:11:27.027829 24259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:44621 every 8 connection(s)
I20260923 15:11:27.028069 24129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:27.028223 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 24129
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176286, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:27.028338 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:27.028570 22321 external_mini_cluster.cc:1428] Setting key 2700adf048a92099edf9350342b441fb
I20260923 15:11:27.032490 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.020978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:53391 (local address 127.21.204.126:37143)
0923 15:11:27.021033 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.021035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.021295 (+ 260us) server_negotiation.cc:407] Connection header received
0923 15:11:27.021986 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.021988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.022002 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.022057 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.022998 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.023948 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.025639 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.025916 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.025994 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.028973 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.028974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.028974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.028982 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.030818 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.031398 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.031399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.031400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.031465 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.031792 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.031793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.031794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.031913 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.031981 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.032331 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.032419 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:11:27.032728 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.020959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:37143 (local address 127.21.204.65:53391)
0923 15:11:27.021156 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.021196 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.021809 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.022144 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.022149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.022574 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:27.022900 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.022908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.024068 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.024070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.025532 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.025536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.026004 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.026005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.026171 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.026758 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:27.026769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:27.028848 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:27.030921 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.030930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.030933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.031278 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.031557 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.031558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.031559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.031697 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.031989 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:27.031996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:27.032168 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.032283 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.032462 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260923 15:11:27.033244 24260 heartbeater.cc:347] Connected to a master server at 127.21.204.126:37143
I20260923 15:11:27.033334 24260 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:27.033509 24260 heartbeater.cc:511] Master 127.21.204.126:37143 requested a full tablet report, sending...
I20260923 15:11:27.034152 24056 ts_manager.cc:194] Registered new tserver with Master: 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621)
I20260923 15:11:27.035167 24056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:53391
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:11:27.040457 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.034375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:33327 (local address 127.21.204.126:37143)
0923 15:11:27.034412 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.034414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.034533 (+ 119us) server_negotiation.cc:407] Connection header received
0923 15:11:27.034605 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.034606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.034616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.034659 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.034995 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.035629 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.036766 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.036951 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.037016 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.038099 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.038100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.038100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.038106 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.039615 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.039953 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.039954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.039955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.040005 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.040124 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.040124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.040125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.040215 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.040259 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.040342 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.040401 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:11:27.040468 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.034327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:37143 (local address 127.21.204.65:33327)
0923 15:11:27.034451 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.034460 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.034547 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.034678 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.034679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.034853 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:27.034902 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.034903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.035744 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.035745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.036668 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.036671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.037036 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.037038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.037123 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.037544 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:27.037550 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:27.037978 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:27.039713 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.039714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.039715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.039887 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.040048 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.040049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.040049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.040090 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.040250 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:27.040251 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:27.040283 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.040328 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.040398 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:27.058430 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:37143
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:27.142381 24269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:27.142576 24269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:27.142601 24269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:27.143781 24269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:27.143823 24269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:27.143844 24269 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 15:11:27.143857 24269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:27.143898 24269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:27.145299 24269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:27.145538 24269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:27.145792 24269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:27.150808 24274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:27.150810 24277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:27.150840 24275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:27.151384 24269 server_base.cc:1034] running on GCE node
I20260923 15:11:27.151613 24269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:27.151822 24269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:27.152997 24269 hybrid_clock.cc:648] HybridClock initialized: now 1790176287152974 us; error 32 us; skew 500 ppm
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:27.155758 24269 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:27.156237 24269 webserver.cc:460] Webserver started at http://127.21.204.66:33203/ using document root <none> and password file <none>
I20260923 15:11:27.156476 24269 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:27.156543 24269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:27.156701 24269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:27.157730 24269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "47eeec232c7e4b1f94994abd14f63442"
format_stamp: "Formatted at 2026-09-23 15:11:27 on dist-test-slave-t5lt"
server_key: "b127f04c94afcac95bf251cc8e95cc5e"
server_key_iv: "5551383ad0174bfd8c7d165af2585b58"
server_key_version: "encryptionkey@0"
I20260923 15:11:27.158116 24269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "47eeec232c7e4b1f94994abd14f63442"
format_stamp: "Formatted at 2026-09-23 15:11:27 on dist-test-slave-t5lt"
server_key: "b127f04c94afcac95bf251cc8e95cc5e"
server_key_iv: "5551383ad0174bfd8c7d165af2585b58"
server_key_version: "encryptionkey@0"
I20260923 15:11:27.159430 24269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:27.160235 24284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.160430 24269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:27.160539 24269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "47eeec232c7e4b1f94994abd14f63442"
format_stamp: "Formatted at 2026-09-23 15:11:27 on dist-test-slave-t5lt"
server_key: "b127f04c94afcac95bf251cc8e95cc5e"
server_key_iv: "5551383ad0174bfd8c7d165af2585b58"
server_key_version: "encryptionkey@0"
I20260923 15:11:27.160622 24269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:27.183161 24269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:27.185462 24269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:27.185688 24269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:27.186364 24269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:27.186436 24269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.186489 24269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:27.186518 24269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:27.194042 24269 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:46095
I20260923 15:11:27.194131 24399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:46095 every 8 connection(s)
I20260923 15:11:27.194432 24269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:27.197002 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.186934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:54691 (local address 127.21.204.126:37143)
0923 15:11:27.186984 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.186986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.187210 (+ 224us) server_negotiation.cc:407] Connection header received
0923 15:11:27.187969 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.187971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.188001 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.188057 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.189038 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.189955 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.191074 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.191450 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.191531 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.193996 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.193997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.193998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.194006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.195559 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.195978 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.195980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.195980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.196045 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.196326 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.196328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.196328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.196459 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.196537 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.196820 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.196908 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:27.197194 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.186922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:37143 (local address 127.21.204.66:54691)
0923 15:11:27.187082 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.187119 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.187761 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.188166 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.188171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.188618 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:27.188928 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.188934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.190054 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.190055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.190976 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.190980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.191625 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.191626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.191799 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.192387 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:27.192396 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:27.193864 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:27.195664 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.195670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.195673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.195897 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.196130 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.196131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.196131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.196253 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.196528 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:27.196534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:27.196688 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.196799 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.196965 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":107,"threads_started":1}
I20260923 15:11:27.197690 24400 heartbeater.cc:347] Connected to a master server at 127.21.204.126:37143
I20260923 15:11:27.197801 24400 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:27.198004 24400 heartbeater.cc:511] Master 127.21.204.126:37143 requested a full tablet report, sending...
I20260923 15:11:27.198483 24057 ts_manager.cc:194] Registered new tserver with Master: 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095)
I20260923 15:11:27.198997 24057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:54691
I20260923 15:11:27.203842 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 24269
I20260923 15:11:27.203956 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:27.204167 22321 external_mini_cluster.cc:1428] Setting key 9b0dda66be85e0e371d87be6a4bfe674
I20260923 15:11:27.204227 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.198745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:37143 (local address 127.21.204.66:54731)
0923 15:11:27.198879 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.198889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.198949 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.199086 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.199087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.199308 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:27.199372 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.199374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.200038 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.200039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.200550 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.200552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.200937 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.200938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.201015 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.201387 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:27.201392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:27.201666 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:27.203015 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.203016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.203016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.203191 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.203410 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.203411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.203411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.203469 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.203860 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:27.203861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:27.203937 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.204016 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.204122 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260923 15:11:27.204247 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.198835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:54731 (local address 127.21.204.126:37143)
0923 15:11:27.198873 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.198875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.198935 (+ 60us) server_negotiation.cc:407] Connection header received
0923 15:11:27.199002 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.199003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.199011 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.199086 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.199474 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.199947 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.200644 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.200851 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.200939 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.201758 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.201759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.201760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.201767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.202917 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.203277 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.203279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.203279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.203333 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.203553 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.203553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.203554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.203772 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.203852 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.204084 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.204162 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:27.235679 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:37143
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:27.321049 24409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:27.321254 24409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:27.321280 24409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:27.322566 24409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:27.322611 24409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:27.322644 24409 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 15:11:27.322656 24409 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:27.322717 24409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:27.324199 24409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:27.324460 24409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:27.324694 24409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:27.329798 24417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:27.329820 24415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:27.329917 24409 server_base.cc:1034] running on GCE node
W20260923 15:11:27.329813 24414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:27.330157 24409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:27.330406 24409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:27.331575 24409 hybrid_clock.cc:648] HybridClock initialized: now 1790176287331557 us; error 52 us; skew 500 ppm
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:27.334331 24409 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:27.334800 24409 webserver.cc:460] Webserver started at http://127.21.204.67:33505/ using document root <none> and password file <none>
I20260923 15:11:27.335029 24409 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:27.335099 24409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:27.335244 24409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:27.336297 24409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "87f9117b32e04c12b62b4364d5b523e4"
format_stamp: "Formatted at 2026-09-23 15:11:27 on dist-test-slave-t5lt"
server_key: "85a528e30efbf554d52e6246020fb2de"
server_key_iv: "876826a10132b490560ec07e497a7e26"
server_key_version: "encryptionkey@0"
I20260923 15:11:27.336660 24409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "87f9117b32e04c12b62b4364d5b523e4"
format_stamp: "Formatted at 2026-09-23 15:11:27 on dist-test-slave-t5lt"
server_key: "85a528e30efbf554d52e6246020fb2de"
server_key_iv: "876826a10132b490560ec07e497a7e26"
server_key_version: "encryptionkey@0"
I20260923 15:11:27.338032 24409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:27.338884 24424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.339238 24409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:27.339356 24409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "87f9117b32e04c12b62b4364d5b523e4"
format_stamp: "Formatted at 2026-09-23 15:11:27 on dist-test-slave-t5lt"
server_key: "85a528e30efbf554d52e6246020fb2de"
server_key_iv: "876826a10132b490560ec07e497a7e26"
server_key_version: "encryptionkey@0"
I20260923 15:11:27.339457 24409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:27.366215 24409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:27.368850 24409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:27.369042 24409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:27.369798 24409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:27.369864 24409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.369935 24409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:27.369962 24409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.378046 24409 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:39255
I20260923 15:11:27.378121 24539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:39255 every 8 connection(s)
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176287, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:27.378463 24409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:27.381311 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 24409
I20260923 15:11:27.381392 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:27.381572 22321 external_mini_cluster.cc:1428] Setting key af8f02c924d1df7eff04486c282598f4
I20260923 15:11:27.381842 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.370495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:37143 (local address 127.21.204.67:53411)
0923 15:11:27.370776 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.370828 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.371613 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.371988 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.371992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.372232 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:27.372415 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.372419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.373467 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.373468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.374451 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.374453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.374892 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.374893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.375050 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.375605 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:27.375624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:27.378660 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:27.380287 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.380294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.380296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.380517 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.380791 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.380792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.380793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.380905 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.381185 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:27.381193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:27.381329 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.381447 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.381585 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
I20260923 15:11:27.382195 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.370376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:53411 (local address 127.21.204.126:37143)
0923 15:11:27.370426 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.370428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.370951 (+ 523us) server_negotiation.cc:407] Connection header received
0923 15:11:27.371775 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.371777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.371792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.371868 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.372535 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.373358 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.374550 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.374769 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.374857 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.378765 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.378767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.378767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.378775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.380162 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.380652 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.380654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.380654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.380717 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.380976 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.380977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.380978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.381103 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.381183 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.382002 (+ 819us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.382094 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:11:27.382515 24540 heartbeater.cc:347] Connected to a master server at 127.21.204.126:37143
I20260923 15:11:27.382601 24540 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:27.382812 24540 heartbeater.cc:511] Master 127.21.204.126:37143 requested a full tablet report, sending...
I20260923 15:11:27.383198 24056 ts_manager.cc:194] Registered new tserver with Master: 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255)
I20260923 15:11:27.383591 24056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:53411
I20260923 15:11:27.388974 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.383467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:37143 (local address 127.21.204.67:36801)
0923 15:11:27.383641 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.383651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.383716 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.383830 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.383831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.384014 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:27.384070 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.384072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.384753 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.384753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.385260 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.385262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.385636 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.385637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.385718 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.386229 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:27.386234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:27.386504 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:27.387911 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.387912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.387913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.388086 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.388329 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.388330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.388330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.388394 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.388637 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:27.388638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:27.388683 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.388781 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.388881 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":126,"threads_started":1}
I20260923 15:11:27.388968 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.383503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:36801 (local address 127.21.204.126:37143)
0923 15:11:27.383540 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.383542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.383698 (+ 156us) server_negotiation.cc:407] Connection header received
0923 15:11:27.383764 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.383765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.383776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.383829 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.384152 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.384648 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.385344 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.385550 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.385633 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.386600 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.386601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.386602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.386608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.387829 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.388173 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.388175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.388175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.388252 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.388459 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.388460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.388460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.388570 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.388646 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.388809 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.388880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 15:11:27.393502 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:27.416563 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.409573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.21.204.126:37143)
0923 15:11:27.409634 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.409636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.409656 (+ 20us) server_negotiation.cc:407] Connection header received
0923 15:11:27.409681 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.409682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.409691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.409753 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.410395 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.411062 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.411803 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.411985 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.412047 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.414241 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.414242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.414243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.414250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.415431 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.415785 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.415788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.415788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.415841 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.416034 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.416035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.416036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.416154 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.416266 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.416375 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.416456 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 15:11:27.418185 24056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:27.419852 24056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:27.426828 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.422341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:39255 (local address 127.0.0.1:59424)
0923 15:11:27.422515 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.422556 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.422623 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.423332 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.423333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.423335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:27.423383 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.423385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.424250 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.424251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.425335 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.425338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.426499 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.426501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.426592 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.426606 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.426664 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.426725 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260923 15:11:27.427040 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.422826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.21.204.65:44621)
0923 15:11:27.423060 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.423064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.423073 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:27.423120 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.423122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.423138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.423214 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:27.424240 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.424897 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.426001 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.426520 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.426590 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.426646 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.426872 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.426917 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260923 15:11:27.427235 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.422577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44621 (local address 127.0.0.1:52940)
0923 15:11:27.422813 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.422823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.422871 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.423286 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.423287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.423289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:27.423349 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.423351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.425035 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.425036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.425813 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.425816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.426689 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.426690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.426741 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.426744 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.426791 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.426828 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260923 15:11:27.427306 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.422639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.21.204.67:39255)
0923 15:11:27.422811 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.422816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.422826 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:27.423216 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.423220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.423237 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.423320 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:27.423454 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.424153 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.425429 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.426393 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.426484 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.426625 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.426768 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.426845 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260923 15:11:27.428432 24194 tablet_service.cc:1467] Processing CreateTablet for tablet 855cfc4df8454ee785516fe660b765c9 (DEFAULT_TABLE table=test-table [id=bccf5623029b41809b0b665e7d8775eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:27.428817 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855cfc4df8454ee785516fe660b765c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.430574 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.422452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.21.204.66:46095)
0923 15:11:27.422621 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.422626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.422801 (+ 175us) server_negotiation.cc:407] Connection header received
0923 15:11:27.423009 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.423012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.423031 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.423146 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:27.423408 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.424148 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.427317 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.427960 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.427999 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.428038 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.430415 (+ 2377us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.430468 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260923 15:11:27.430588 24474 tablet_service.cc:1467] Processing CreateTablet for tablet 855cfc4df8454ee785516fe660b765c9 (DEFAULT_TABLE table=test-table [id=bccf5623029b41809b0b665e7d8775eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:27.430804 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.422357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46095 (local address 127.0.0.1:48538)
0923 15:11:27.422713 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.422723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.422879 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.423159 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.423162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.423164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:27.423280 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.423282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.425450 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.425451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.426828 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.426837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.430225 (+ 3388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.430226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.430306 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.430310 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.430672 (+ 362us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.430728 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":99,"threads_started":1}
I20260923 15:11:27.430965 24474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855cfc4df8454ee785516fe660b765c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.431574 24334 tablet_service.cc:1467] Processing CreateTablet for tablet 855cfc4df8454ee785516fe660b765c9 (DEFAULT_TABLE table=test-table [id=bccf5623029b41809b0b665e7d8775eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:27.431964 24334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855cfc4df8454ee785516fe660b765c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.432631 24563 tablet_bootstrap.cc:492] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Bootstrap starting.
I20260923 15:11:27.433357 24563 tablet_bootstrap.cc:654] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.433696 24563 log.cc:824] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:27.434111 24564 tablet_bootstrap.cc:492] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Bootstrap starting.
I20260923 15:11:27.434796 24564 tablet_bootstrap.cc:654] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.435091 24564 log.cc:824] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:27.435278 24566 tablet_bootstrap.cc:492] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Bootstrap starting.
I20260923 15:11:27.435767 24563 tablet_bootstrap.cc:492] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: No bootstrap required, opened a new log
I20260923 15:11:27.435840 24564 tablet_bootstrap.cc:492] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: No bootstrap required, opened a new log
I20260923 15:11:27.435853 24563 ts_tablet_manager.cc:1397] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 15:11:27.435920 24564 ts_tablet_manager.cc:1397] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:27.435931 24566 tablet_bootstrap.cc:654] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.436249 24566 log.cc:824] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:27.437041 24566 tablet_bootstrap.cc:492] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: No bootstrap required, opened a new log
I20260923 15:11:27.437112 24566 ts_tablet_manager.cc:1397] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:27.437515 24563 raft_consensus.cc:348] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.437670 24563 raft_consensus.cc:374] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.437712 24563 raft_consensus.cc:729] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7974ed8621194b3fa1128f5e200db1ac, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.437827 24563 consensus_queue.cc:260] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.437952 24564 raft_consensus.cc:348] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.438136 24563 ts_tablet_manager.cc:1428] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 15:11:27.438226 24260 heartbeater.cc:503] Master 127.21.204.126:37143 was elected leader, sending a full tablet report...
I20260923 15:11:27.438498 24564 raft_consensus.cc:374] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.438560 24564 raft_consensus.cc:729] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f9117b32e04c12b62b4364d5b523e4, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.438665 24564 consensus_queue.cc:260] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.438949 24564 ts_tablet_manager.cc:1428] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:27.439004 24540 heartbeater.cc:503] Master 127.21.204.126:37143 was elected leader, sending a full tablet report...
I20260923 15:11:27.440053 24566 raft_consensus.cc:348] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.440183 24566 raft_consensus.cc:374] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.440203 24566 raft_consensus.cc:729] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eeec232c7e4b1f94994abd14f63442, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.440330 24566 consensus_queue.cc:260] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.440605 24566 ts_tablet_manager.cc:1428] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:27.440619 24400 heartbeater.cc:503] Master 127.21.204.126:37143 was elected leader, sending a full tablet report...
W20260923 15:11:27.445289 24401 tablet.cc:2369] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:27.529100 24261 tablet.cc:2369] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:27.531445 24569 raft_consensus.cc:482] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:27.531546 24569 raft_consensus.cc:504] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.531908 24569 leader_election.cc:288] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095), 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255)
I20260923 15:11:27.536108 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.532224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:53259 (local address 127.21.204.67:39255)
0923 15:11:27.532271 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.532274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.532384 (+ 110us) server_negotiation.cc:407] Connection header received
0923 15:11:27.532454 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.532455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.532464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.532543 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:27.532687 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.533419 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.534965 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.535530 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.535633 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.535643 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.535991 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.536038 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:11:27.536108 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.532247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:38971 (local address 127.21.204.66:46095)
0923 15:11:27.532297 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.532299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.532308 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:27.532343 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.532344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.532351 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.532418 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:27.532650 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.533408 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.534965 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.535576 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.535643 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.535652 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.535991 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.536041 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:27.536104 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.532109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46095 (local address 127.21.204.65:38971)
0923 15:11:27.532187 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.532201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.532249 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.532422 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.532423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.532425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:27.532503 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.532505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.533503 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.533504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.534834 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.534837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.535766 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.535767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.535870 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.535877 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.535945 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.535999 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":78}
I20260923 15:11:27.536123 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.532152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:39255 (local address 127.21.204.65:53259)
0923 15:11:27.532300 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.532310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.532397 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.532563 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.532564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.532565 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:27.532628 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.532629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.533503 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.533504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.534834 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.534837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.535786 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.535786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.535870 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.535877 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.535943 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.535999 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":39,"thread_start_us":78,"threads_started":1}
I20260923 15:11:27.536556 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855cfc4df8454ee785516fe660b765c9" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442" is_pre_election: true
I20260923 15:11:27.536726 24354 raft_consensus.cc:2454] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 0.
I20260923 15:11:27.536856 24494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855cfc4df8454ee785516fe660b765c9" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9117b32e04c12b62b4364d5b523e4" is_pre_election: true
I20260923 15:11:27.537015 24494 raft_consensus.cc:2454] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 0.
I20260923 15:11:27.536995 24148 leader_election.cc:302] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eeec232c7e4b1f94994abd14f63442, 7974ed8621194b3fa1128f5e200db1ac; no voters:
I20260923 15:11:27.537132 24569 raft_consensus.cc:2781] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:27.537212 24569 raft_consensus.cc:482] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:27.537248 24569 raft_consensus.cc:3037] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:27.538127 24569 raft_consensus.cc:504] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.538277 24569 leader_election.cc:288] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 election: Requested vote from peers 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095), 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255)
I20260923 15:11:27.538424 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855cfc4df8454ee785516fe660b765c9" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442"
I20260923 15:11:27.538501 24354 raft_consensus.cc:3037] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:27.538522 24494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855cfc4df8454ee785516fe660b765c9" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9117b32e04c12b62b4364d5b523e4"
I20260923 15:11:27.538604 24494 raft_consensus.cc:3037] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:27.539198 24354 raft_consensus.cc:2454] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 1.
I20260923 15:11:27.539289 24494 raft_consensus.cc:2454] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 1.
I20260923 15:11:27.539382 24148 leader_election.cc:302] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eeec232c7e4b1f94994abd14f63442, 7974ed8621194b3fa1128f5e200db1ac; no voters:
I20260923 15:11:27.539479 24569 raft_consensus.cc:2781] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:27.539638 24569 raft_consensus.cc:686] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 1 LEADER]: Becoming Leader. State: Replica: 7974ed8621194b3fa1128f5e200db1ac, State: Running, Role: LEADER
I20260923 15:11:27.539786 24569 consensus_queue.cc:237] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } }
I20260923 15:11:27.540628 24056 catalog_manager.cc:5495] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65). New cstate: current_term: 1 leader_uuid: "7974ed8621194b3fa1128f5e200db1ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } health_report { overall_health: HEALTHY } } }
Sep 23 15:11:27 dist-test-slave-t5lt krb5kdc[24012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176286, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:27.594262 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.586755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:37143 (local address 127.0.0.1:58354)
0923 15:11:27.586947 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.586958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.587026 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.587156 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.587157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.587455 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:27.587506 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.587508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.588398 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.588399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.588939 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.588940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.589313 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.589313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.589353 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.589934 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:27.589939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:27.591253 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:27.592907 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.592915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.592917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.593144 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.593397 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:27.593398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:27.593398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:27.593461 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:27.593814 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:27.593822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:27.593935 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.594022 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.594134 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
I20260923 15:11:27.594266 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.586888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.21.204.126:37143)
0923 15:11:27.586930 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.586932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.587024 (+ 92us) server_negotiation.cc:407] Connection header received
0923 15:11:27.587074 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.587075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.587084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.587134 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:27.587600 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.588316 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.589019 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.589240 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.589302 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.591365 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:27.591366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:27.591366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:27.591372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:27.592792 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.593234 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.593235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.593235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.593296 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:27.593543 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:27.593544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:27.593545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:27.593714 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:27.593774 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.594063 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.594183 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":22}
I20260923 15:11:27.596858 24056 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:58354:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 15:11:27.597088 24056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:27.598814 24334 tablet_service.cc:1467] Processing CreateTablet for tablet 83ddfb9a7cdb43e1938b06274ce1b8eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fff1bd97340347ada3cd476b7b6239a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:27.598848 24474 tablet_service.cc:1467] Processing CreateTablet for tablet 83ddfb9a7cdb43e1938b06274ce1b8eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fff1bd97340347ada3cd476b7b6239a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:27.598851 24194 tablet_service.cc:1467] Processing CreateTablet for tablet 83ddfb9a7cdb43e1938b06274ce1b8eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fff1bd97340347ada3cd476b7b6239a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:27.598999 24474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ddfb9a7cdb43e1938b06274ce1b8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.598999 24334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ddfb9a7cdb43e1938b06274ce1b8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.599006 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ddfb9a7cdb43e1938b06274ce1b8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.600123 24563 tablet_bootstrap.cc:492] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: Bootstrap starting.
I20260923 15:11:27.600473 24566 tablet_bootstrap.cc:492] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: Bootstrap starting.
I20260923 15:11:27.600534 24563 tablet_bootstrap.cc:654] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.600574 24564 tablet_bootstrap.cc:492] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: Bootstrap starting.
I20260923 15:11:27.601061 24566 tablet_bootstrap.cc:654] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.601158 24564 tablet_bootstrap.cc:654] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.601579 24563 tablet_bootstrap.cc:492] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: No bootstrap required, opened a new log
I20260923 15:11:27.601645 24563 ts_tablet_manager.cc:1397] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260923 15:11:27.601807 24564 tablet_bootstrap.cc:492] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: No bootstrap required, opened a new log
I20260923 15:11:27.601854 24564 ts_tablet_manager.cc:1397] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:27.601814 24563 raft_consensus.cc:348] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.601923 24563 raft_consensus.cc:374] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.601953 24563 raft_consensus.cc:729] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7974ed8621194b3fa1128f5e200db1ac, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.602008 24563 consensus_queue.cc:260] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.602049 24566 tablet_bootstrap.cc:492] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: No bootstrap required, opened a new log
I20260923 15:11:27.602088 24566 ts_tablet_manager.cc:1397] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:27.602037 24564 raft_consensus.cc:348] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.602105 24563 ts_tablet_manager.cc:1428] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.602114 24564 raft_consensus.cc:374] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.602137 24564 raft_consensus.cc:729] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f9117b32e04c12b62b4364d5b523e4, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.602190 24564 consensus_queue.cc:260] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.602233 24569 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.602269 24564 ts_tablet_manager.cc:1428] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.602249 24566 raft_consensus.cc:348] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.602309 24566 raft_consensus.cc:374] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.602300 24569 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:27.602331 24566 raft_consensus.cc:729] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eeec232c7e4b1f94994abd14f63442, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.602375 24566 consensus_queue.cc:260] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.602406 24570 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.602473 24566 ts_tablet_manager.cc:1428] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.602486 24570 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:27.602691 24571 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.602794 24571 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 15:11:27.629163 24541 tablet.cc:2369] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:27.689321 24570 raft_consensus.cc:482] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:27.689441 24570 raft_consensus.cc:504] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.689954 24570 leader_election.cc:288] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621), 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095)
I20260923 15:11:27.694020 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.690271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:51727 (local address 127.21.204.65:44621)
0923 15:11:27.690488 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.690490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.690499 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:27.690528 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.690528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.690538 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.690586 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:27.690787 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.691521 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.692936 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.693596 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.693679 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.693691 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.693905 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.693948 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260923 15:11:27.694036 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.690187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44621 (local address 127.21.204.67:51727)
0923 15:11:27.690330 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.690343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.690410 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.690642 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.690643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.690645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:27.690720 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.690722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.691640 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.691641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.692856 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.692858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.693690 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.693690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.693799 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.693803 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.693880 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.693949 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260923 15:11:27.694589 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.690305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46095 (local address 127.21.204.67:52743)
0923 15:11:27.690464 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.690473 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.690557 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.690683 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.690684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.690685 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:27.690755 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.690757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.691784 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.691785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.692856 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.692858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.694405 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.694406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.694460 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.694462 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.694507 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.694535 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":8,"thread_start_us":95,"threads_started":1}
I20260923 15:11:27.694653 24214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddfb9a7cdb43e1938b06274ce1b8eb" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7974ed8621194b3fa1128f5e200db1ac" is_pre_election: true
I20260923 15:11:27.694818 24214 raft_consensus.cc:2454] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 0.
I20260923 15:11:27.695103 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.690373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:52743 (local address 127.21.204.66:46095)
0923 15:11:27.690420 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.690422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.690549 (+ 127us) server_negotiation.cc:407] Connection header received
0923 15:11:27.690616 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.690616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.690626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.690683 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:27.690859 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.691682 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.693730 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.694345 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.694402 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.694409 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.694958 (+ 549us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.695012 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:27.695328 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddfb9a7cdb43e1938b06274ce1b8eb" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442" is_pre_election: true
I20260923 15:11:27.695447 24354 raft_consensus.cc:2454] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 0.
I20260923 15:11:27.695627 24428 leader_election.cc:302] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eeec232c7e4b1f94994abd14f63442, 87f9117b32e04c12b62b4364d5b523e4; no voters:
I20260923 15:11:27.696048 24570 raft_consensus.cc:2781] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:27.696111 24570 raft_consensus.cc:482] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:27.696138 24570 raft_consensus.cc:3037] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:27.696852 24570 raft_consensus.cc:504] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.696993 24570 leader_election.cc:288] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 election: Requested vote from peers 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621), 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095)
I20260923 15:11:27.697260 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddfb9a7cdb43e1938b06274ce1b8eb" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442"
I20260923 15:11:27.697341 24354 raft_consensus.cc:3037] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:27.697352 24214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddfb9a7cdb43e1938b06274ce1b8eb" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7974ed8621194b3fa1128f5e200db1ac"
I20260923 15:11:27.697464 24214 raft_consensus.cc:3037] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:27.698091 24354 raft_consensus.cc:2454] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 1.
I20260923 15:11:27.698342 24428 leader_election.cc:302] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eeec232c7e4b1f94994abd14f63442, 87f9117b32e04c12b62b4364d5b523e4; no voters:
I20260923 15:11:27.698460 24570 raft_consensus.cc:2781] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:27.698621 24570 raft_consensus.cc:686] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [term 1 LEADER]: Becoming Leader. State: Replica: 87f9117b32e04c12b62b4364d5b523e4, State: Running, Role: LEADER
I20260923 15:11:27.698727 24570 consensus_queue.cc:237] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.699447 24585 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: TxnStatusTablet state changed. Reason: New leader 87f9117b32e04c12b62b4364d5b523e4. Latest consensus state: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.699532 24585 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:27.699564 24057 catalog_manager.cc:5495] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67). New cstate: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } health_report { overall_health: HEALTHY } } }
I20260923 15:11:27.699693 24214 raft_consensus.cc:2454] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 1.
I20260923 15:11:27.699733 24587 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 15:11:27.699822 24587 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 15:11:27.721793 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.718993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:39255 (local address 127.0.0.1:59426)
0923 15:11:27.719075 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.719089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.719142 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.719334 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.719336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.719337 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:27.719395 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.719397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.720110 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.720112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.720758 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.720760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.721186 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.721187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.721257 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.721286 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:27.721601 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:27.721614 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.721656 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.721710 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 15:11:27.721787 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.719099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.21.204.67:39255)
0923 15:11:27.719154 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.719156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.719167 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:11:27.719232 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.719233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.719254 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.719323 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:27.719483 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.720007 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.720873 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.721102 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.721187 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.721406 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:27.721523 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:27.721626 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.721656 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.721680 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 15:11:27.723855 24354 raft_consensus.cc:1261] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Refusing update from remote peer 87f9117b32e04c12b62b4364d5b523e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:27.723875 24214 raft_consensus.cc:1261] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Refusing update from remote peer 87f9117b32e04c12b62b4364d5b523e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:27.724269 24585 consensus_queue.cc:1035] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:27.724424 24570 consensus_queue.cc:1035] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:27.725782 24569 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: TxnStatusTablet state changed. Reason: New leader 87f9117b32e04c12b62b4364d5b523e4. Latest consensus state: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.725893 24569 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:27.725817 24571 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: TxnStatusTablet state changed. Reason: New leader 87f9117b32e04c12b62b4364d5b523e4. Latest consensus state: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.725961 24571 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:27.727185 24585 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.727279 24585 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:27.727326 24585 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.727363 24585 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 87f9117b32e04c12b62b4364d5b523e4: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:27.727322 24569 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.727401 24569 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 7974ed8621194b3fa1128f5e200db1ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:27.727521 24590 mvcc.cc:205] Tried to move back new op lower bound from 7332562074514026496 to 7332562074414465024. Current Snapshot: MvccSnapshot[applied={T|T < 7332562074514026496 or (T in {7332562074514026496})}]
I20260923 15:11:27.727676 24591 mvcc.cc:205] Tried to move back new op lower bound from 7332562074514026496 to 7332562074414465024. Current Snapshot: MvccSnapshot[applied={T|T < 7332562074514026496}]
I20260923 15:11:27.728818 24571 tablet_replica.cc:439] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } } }
I20260923 15:11:27.728911 24571 tablet_replica.cc:442] T 83ddfb9a7cdb43e1938b06274ce1b8eb P 47eeec232c7e4b1f94994abd14f63442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:27.729017 24592 mvcc.cc:205] Tried to move back new op lower bound from 7332562074514026496 to 7332562074414465024. Current Snapshot: MvccSnapshot[applied={T|T < 7332562074514026496}]
I20260923 15:11:27.732244 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.729307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.21.204.65:44621)
0923 15:11:27.729359 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.729361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.729442 (+ 81us) server_negotiation.cc:407] Connection header received
0923 15:11:27.729525 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.729525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.729544 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.729643 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:27.729795 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.730356 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.731498 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.731703 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.731762 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.731934 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:27.732025 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:27.732082 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.732147 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.732164 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:11:27.737852 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.734862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:37617 (local address 127.21.204.67:39255)
0923 15:11:27.734914 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.734916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.735017 (+ 101us) server_negotiation.cc:407] Connection header received
0923 15:11:27.735065 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.735066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.735076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.735143 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:27.735368 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.736119 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.736956 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.737266 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.737339 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.737528 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:27.737558 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:27.737612 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.737756 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.737776 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:11:27.737857 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.734747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:39255 (local address 127.21.204.65:37617)
0923 15:11:27.734924 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.734936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.735009 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.735182 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.735183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.735184 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:27.735242 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.735243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.736237 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.736238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.736872 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.736874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.737358 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.737360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.737416 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.737431 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:27.737653 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:27.737656 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.737747 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.737785 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260923 15:11:27.740576 24354 raft_consensus.cc:1261] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Refusing update from remote peer 7974ed8621194b3fa1128f5e200db1ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:27.740662 24494 raft_consensus.cc:1261] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Refusing update from remote peer 7974ed8621194b3fa1128f5e200db1ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:27.740893 24573 consensus_queue.cc:1035] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:27.741052 24569 consensus_queue.cc:1035] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:27.749467 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.746630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44621 (local address 127.21.204.67:43419)
0923 15:11:27.746688 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:27.746699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:27.746741 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:27.746908 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:27.746909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:27.746910 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:27.746978 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.746980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.747847 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.747848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.748553 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:27.748556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.749030 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.749031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:27.749079 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.749092 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:27.749292 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:27.749294 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:27.749348 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 15:11:27.749393 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 15:11:27.749467 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:27.746735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:43419 (local address 127.21.204.65:44621)
0923 15:11:27.746769 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:27.746771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:27.746779 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:27.746800 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:27.746801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:27.746811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:27.746863 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:27.747050 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.747768 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.748629 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:27.748958 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:27.749029 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:27.749178 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:27.749211 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:27.749291 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:27.749386 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 15:11:27.749405 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 15:11:27.759402 24057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:27.759512 24057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:27.760303 24057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Sending DeleteTablet for 3 replicas of tablet 855cfc4df8454ee785516fe660b765c9
I20260923 15:11:27.760664 24334 tablet_service.cc:1514] Processing DeleteTablet for tablet 855cfc4df8454ee785516fe660b765c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:27 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:48538
I20260923 15:11:27.760968 24474 tablet_service.cc:1514] Processing DeleteTablet for tablet 855cfc4df8454ee785516fe660b765c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:27 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:59424
I20260923 15:11:27.761175 24057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:27.761339 24057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:27.761722 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet 855cfc4df8454ee785516fe660b765c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:27 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:52940
I20260923 15:11:27.761957 24611 tablet_replica.cc:330] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: stopping tablet replica
I20260923 15:11:27.762110 24611 raft_consensus.cc:2229] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:27.762247 24611 raft_consensus.cc:2258] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:27.762905 24194 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca153f9e2104aaab9a84cae0f33cc82 (DEFAULT_TABLE table=test-table [id=3fcf10d5136e4359a8ee5cd529511a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:27.763029 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca153f9e2104aaab9a84cae0f33cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.763621 24334 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca153f9e2104aaab9a84cae0f33cc82 (DEFAULT_TABLE table=test-table [id=3fcf10d5136e4359a8ee5cd529511a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:27.764896 24334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca153f9e2104aaab9a84cae0f33cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.766297 24566 tablet_bootstrap.cc:492] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: Bootstrap starting.
I20260923 15:11:27.766739 24566 tablet_bootstrap.cc:654] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.767711 24563 tablet_bootstrap.cc:492] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: Bootstrap starting.
I20260923 15:11:27.767742 24474 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca153f9e2104aaab9a84cae0f33cc82 (DEFAULT_TABLE table=test-table [id=3fcf10d5136e4359a8ee5cd529511a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:27.767850 24474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca153f9e2104aaab9a84cae0f33cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:27.768134 24563 tablet_bootstrap.cc:654] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.769174 24564 tablet_bootstrap.cc:492] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: Bootstrap starting.
I20260923 15:11:27.769536 24564 tablet_bootstrap.cc:654] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:27.769784 24611 ts_tablet_manager.cc:1905] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:27.771433 24609 tablet_replica.cc:330] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: stopping tablet replica
I20260923 15:11:27.771605 24611 ts_tablet_manager.cc:1918] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:27.771677 24611 log.cc:1196] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/855cfc4df8454ee785516fe660b765c9
I20260923 15:11:27.771939 24611 ts_tablet_manager.cc:1939] T 855cfc4df8454ee785516fe660b765c9 P 7974ed8621194b3fa1128f5e200db1ac: Deleting consensus metadata
I20260923 15:11:27.772353 24610 tablet_replica.cc:330] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: stopping tablet replica
I20260923 15:11:27.772384 24044 catalog_manager.cc:4841] TS 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621): tablet 855cfc4df8454ee785516fe660b765c9 (table test-table [id=bccf5623029b41809b0b665e7d8775eb]) successfully deleted
I20260923 15:11:27.772527 24610 raft_consensus.cc:2229] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:27.772598 24610 raft_consensus.cc:2258] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:27.772732 24609 raft_consensus.cc:2229] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:27.772812 24609 raft_consensus.cc:2258] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:27.773276 24610 ts_tablet_manager.cc:1905] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:27.774264 24563 tablet_bootstrap.cc:492] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: No bootstrap required, opened a new log
I20260923 15:11:27.774264 24566 tablet_bootstrap.cc:492] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: No bootstrap required, opened a new log
I20260923 15:11:27.774313 24563 ts_tablet_manager.cc:1397] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260923 15:11:27.774313 24566 ts_tablet_manager.cc:1397] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260923 15:11:27.774461 24566 raft_consensus.cc:348] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.774456 24563 raft_consensus.cc:348] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.774524 24563 raft_consensus.cc:374] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.774524 24566 raft_consensus.cc:374] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.774546 24563 raft_consensus.cc:729] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7974ed8621194b3fa1128f5e200db1ac, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.774546 24566 raft_consensus.cc:729] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eeec232c7e4b1f94994abd14f63442, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.774587 24563 consensus_queue.cc:260] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.774591 24566 consensus_queue.cc:260] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.774677 24563 ts_tablet_manager.cc:1428] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.774691 24566 ts_tablet_manager.cc:1428] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:27.774731 24610 ts_tablet_manager.cc:1918] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:27.774793 24610 log.cc:1196] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/855cfc4df8454ee785516fe660b765c9
I20260923 15:11:27.775048 24609 ts_tablet_manager.cc:1905] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:27.776876 24609 ts_tablet_manager.cc:1918] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:27.776952 24609 log.cc:1196] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/855cfc4df8454ee785516fe660b765c9
I20260923 15:11:27.777209 24609 ts_tablet_manager.cc:1939] T 855cfc4df8454ee785516fe660b765c9 P 47eeec232c7e4b1f94994abd14f63442: Deleting consensus metadata
I20260923 15:11:27.775689 24564 tablet_bootstrap.cc:492] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: No bootstrap required, opened a new log
I20260923 15:11:27.777402 24564 ts_tablet_manager.cc:1397] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260923 15:11:27.777539 24564 raft_consensus.cc:348] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.777590 24610 ts_tablet_manager.cc:1939] T 855cfc4df8454ee785516fe660b765c9 P 87f9117b32e04c12b62b4364d5b523e4: Deleting consensus metadata
I20260923 15:11:27.777611 24564 raft_consensus.cc:374] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:27.777631 24564 raft_consensus.cc:729] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f9117b32e04c12b62b4364d5b523e4, State: Initialized, Role: FOLLOWER
I20260923 15:11:27.777674 24564 consensus_queue.cc:260] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:27.777756 24564 ts_tablet_manager.cc:1428] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:27.778069 24043 catalog_manager.cc:4841] TS 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255): tablet 855cfc4df8454ee785516fe660b765c9 (table test-table [id=bccf5623029b41809b0b665e7d8775eb]) successfully deleted
I20260923 15:11:27.778571 24044 catalog_manager.cc:4841] TS 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095): tablet 855cfc4df8454ee785516fe660b765c9 (table test-table [id=bccf5623029b41809b0b665e7d8775eb]) successfully deleted
I20260923 15:11:28.159653 24605 raft_consensus.cc:482] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:28.159750 24605 raft_consensus.cc:504] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:28.159942 24605 leader_election.cc:288] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095), 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255)
I20260923 15:11:28.160234 24494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca153f9e2104aaab9a84cae0f33cc82" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9117b32e04c12b62b4364d5b523e4" is_pre_election: true
I20260923 15:11:28.160234 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca153f9e2104aaab9a84cae0f33cc82" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442" is_pre_election: true
I20260923 15:11:28.160351 24354 raft_consensus.cc:2454] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 0.
I20260923 15:11:28.160352 24494 raft_consensus.cc:2454] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 0.
I20260923 15:11:28.160523 24147 leader_election.cc:302] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7974ed8621194b3fa1128f5e200db1ac, 87f9117b32e04c12b62b4364d5b523e4; no voters:
I20260923 15:11:28.160651 24605 raft_consensus.cc:2781] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:28.160705 24605 raft_consensus.cc:482] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:28.160734 24605 raft_consensus.cc:3037] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:28.161444 24605 raft_consensus.cc:504] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:28.161623 24605 leader_election.cc:288] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 election: Requested vote from peers 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095), 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255)
I20260923 15:11:28.161782 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca153f9e2104aaab9a84cae0f33cc82" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442"
I20260923 15:11:28.161800 24494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca153f9e2104aaab9a84cae0f33cc82" candidate_uuid: "7974ed8621194b3fa1128f5e200db1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9117b32e04c12b62b4364d5b523e4"
I20260923 15:11:28.161911 24354 raft_consensus.cc:3037] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:28.161921 24494 raft_consensus.cc:3037] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:28.162523 24494 raft_consensus.cc:2454] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 1.
I20260923 15:11:28.162645 24354 raft_consensus.cc:2454] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7974ed8621194b3fa1128f5e200db1ac in term 1.
I20260923 15:11:28.162701 24147 leader_election.cc:302] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7974ed8621194b3fa1128f5e200db1ac, 87f9117b32e04c12b62b4364d5b523e4; no voters:
I20260923 15:11:28.162817 24605 raft_consensus.cc:2781] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:28.162887 24605 raft_consensus.cc:686] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 1 LEADER]: Becoming Leader. State: Replica: 7974ed8621194b3fa1128f5e200db1ac, State: Running, Role: LEADER
I20260923 15:11:28.162932 24605 consensus_queue.cc:237] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:28.163630 24057 catalog_manager.cc:5495] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65). New cstate: current_term: 1 leader_uuid: "7974ed8621194b3fa1128f5e200db1ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:28.214195 24354 raft_consensus.cc:1261] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Refusing update from remote peer 7974ed8621194b3fa1128f5e200db1ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:28.214252 24494 raft_consensus.cc:1261] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Refusing update from remote peer 7974ed8621194b3fa1128f5e200db1ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:28.214464 24569 consensus_queue.cc:1035] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:28.214571 24605 consensus_queue.cc:1035] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 15:11:28.219058 24591 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 15:11:29.737107 24055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:29.737211 24055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:29.738078 24055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Sending DeleteTablet for 3 replicas of tablet 0ca153f9e2104aaab9a84cae0f33cc82
I20260923 15:11:29.738397 24474 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca153f9e2104aaab9a84cae0f33cc82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:29 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:59424
I20260923 15:11:29.738387 24334 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca153f9e2104aaab9a84cae0f33cc82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:29 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:48538
I20260923 15:11:29.738607 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca153f9e2104aaab9a84cae0f33cc82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:29 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:52940
I20260923 15:11:29.738922 24642 tablet_replica.cc:330] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: stopping tablet replica
I20260923 15:11:29.738981 24642 raft_consensus.cc:2229] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:29.739022 24642 raft_consensus.cc:2258] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:29.739244 24642 ts_tablet_manager.cc:1905] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:29.740281 24641 tablet_replica.cc:330] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: stopping tablet replica
I20260923 15:11:29.740386 24641 raft_consensus.cc:2229] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:29.740489 24641 raft_consensus.cc:2258] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:29.740933 24641 ts_tablet_manager.cc:1905] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:29.741080 24642 ts_tablet_manager.cc:1918] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 15:11:29.741207 24642 log.cc:1196] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/0ca153f9e2104aaab9a84cae0f33cc82
I20260923 15:11:29.741468 24642 ts_tablet_manager.cc:1939] T 0ca153f9e2104aaab9a84cae0f33cc82 P 7974ed8621194b3fa1128f5e200db1ac: Deleting consensus metadata
I20260923 15:11:29.741744 24640 tablet_replica.cc:330] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: stopping tablet replica
I20260923 15:11:29.741811 24044 catalog_manager.cc:4841] TS 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621): tablet 0ca153f9e2104aaab9a84cae0f33cc82 (table test-table [id=3fcf10d5136e4359a8ee5cd529511a0b]) successfully deleted
I20260923 15:11:29.741850 24640 raft_consensus.cc:2229] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:29.741917 24640 raft_consensus.cc:2258] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:29.742169 24640 ts_tablet_manager.cc:1905] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:29.742583 24641 ts_tablet_manager.cc:1918] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 15:11:29.742660 24641 log.cc:1196] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/0ca153f9e2104aaab9a84cae0f33cc82
I20260923 15:11:29.742856 24641 ts_tablet_manager.cc:1939] T 0ca153f9e2104aaab9a84cae0f33cc82 P 47eeec232c7e4b1f94994abd14f63442: Deleting consensus metadata
I20260923 15:11:29.743251 24044 catalog_manager.cc:4841] TS 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095): tablet 0ca153f9e2104aaab9a84cae0f33cc82 (table test-table [id=3fcf10d5136e4359a8ee5cd529511a0b]) successfully deleted
I20260923 15:11:29.743690 24640 ts_tablet_manager.cc:1918] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 15:11:29.743736 24640 log.cc:1196] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/0ca153f9e2104aaab9a84cae0f33cc82
I20260923 15:11:29.743947 24640 ts_tablet_manager.cc:1939] T 0ca153f9e2104aaab9a84cae0f33cc82 P 87f9117b32e04c12b62b4364d5b523e4: Deleting consensus metadata
I20260923 15:11:29.744292 24043 catalog_manager.cc:4841] TS 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255): tablet 0ca153f9e2104aaab9a84cae0f33cc82 (table test-table [id=3fcf10d5136e4359a8ee5cd529511a0b]) successfully deleted
Sep 23 15:11:29 dist-test-slave-t5lt krb5kdc[24012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176289, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 15:11:29 dist-test-slave-t5lt krb5kdc[24012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176289, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:29.769027 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.756683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.21.204.126:37143)
0923 15:11:29.756817 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:29.756819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:29.756828 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:29.756871 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:29.756872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:29.756883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:29.756942 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:29.757512 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.758078 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.758830 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.759015 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.759064 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.766465 (+ 7401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:29.766471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:29.766471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:29.766478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:29.767853 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:29.768229 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:29.768232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:29.768232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:29.768289 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:29.768488 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:29.768489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:29.768489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:29.768609 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:29.768726 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:29.768824 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 15:11:29.768915 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260923 15:11:29.771204 24055 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:29.771390 24055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:29.772894 24474 tablet_service.cc:1467] Processing CreateTablet for tablet dd0f80a4aee24883bdc1367ec79c3f37 (DEFAULT_TABLE table=test-table [id=a3d286436a484353a189b6e35a6234d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:29.772871 24194 tablet_service.cc:1467] Processing CreateTablet for tablet dd0f80a4aee24883bdc1367ec79c3f37 (DEFAULT_TABLE table=test-table [id=a3d286436a484353a189b6e35a6234d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:29.772989 24334 tablet_service.cc:1467] Processing CreateTablet for tablet dd0f80a4aee24883bdc1367ec79c3f37 (DEFAULT_TABLE table=test-table [id=a3d286436a484353a189b6e35a6234d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:29.773032 24474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0f80a4aee24883bdc1367ec79c3f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:29.773047 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0f80a4aee24883bdc1367ec79c3f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:29.773119 24334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd0f80a4aee24883bdc1367ec79c3f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:29.774219 24651 tablet_bootstrap.cc:492] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: Bootstrap starting.
I20260923 15:11:29.774709 24652 tablet_bootstrap.cc:492] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: Bootstrap starting.
I20260923 15:11:29.774914 24651 tablet_bootstrap.cc:654] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:29.774948 24653 tablet_bootstrap.cc:492] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: Bootstrap starting.
I20260923 15:11:29.775067 24652 tablet_bootstrap.cc:654] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:29.775355 24653 tablet_bootstrap.cc:654] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:29.775820 24652 tablet_bootstrap.cc:492] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: No bootstrap required, opened a new log
I20260923 15:11:29.775899 24652 ts_tablet_manager.cc:1397] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:29.776044 24653 tablet_bootstrap.cc:492] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: No bootstrap required, opened a new log
I20260923 15:11:29.776022 24652 raft_consensus.cc:348] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.776091 24653 ts_tablet_manager.cc:1397] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:29.776098 24652 raft_consensus.cc:374] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:29.776124 24652 raft_consensus.cc:729] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7974ed8621194b3fa1128f5e200db1ac, State: Initialized, Role: FOLLOWER
I20260923 15:11:29.776180 24652 consensus_queue.cc:260] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.776242 24653 raft_consensus.cc:348] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.776302 24652 ts_tablet_manager.cc:1428] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:29.776338 24653 raft_consensus.cc:374] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:29.776362 24653 raft_consensus.cc:729] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eeec232c7e4b1f94994abd14f63442, State: Initialized, Role: FOLLOWER
I20260923 15:11:29.776407 24653 consensus_queue.cc:260] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.776589 24653 ts_tablet_manager.cc:1428] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:29.776679 24651 tablet_bootstrap.cc:492] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: No bootstrap required, opened a new log
I20260923 15:11:29.776726 24651 ts_tablet_manager.cc:1397] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 15:11:29.776829 24651 raft_consensus.cc:348] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.776887 24651 raft_consensus.cc:374] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:29.776907 24651 raft_consensus.cc:729] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f9117b32e04c12b62b4364d5b523e4, State: Initialized, Role: FOLLOWER
I20260923 15:11:29.776957 24651 consensus_queue.cc:260] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.777060 24651 ts_tablet_manager.cc:1428] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:29.832777 24657 raft_consensus.cc:482] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:29.832907 24657 raft_consensus.cc:504] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.833377 24657 leader_election.cc:288] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621), 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255)
I20260923 15:11:29.836675 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.833676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:45147 (local address 127.21.204.65:44621)
0923 15:11:29.833821 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:29.833823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:29.833835 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:11:29.833952 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:29.833953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:29.833963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:29.834033 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:29.834210 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.834785 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.835912 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.836307 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.836396 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.836406 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:29.836569 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 15:11:29.836611 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260923 15:11:29.836964 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.833532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44621 (local address 127.21.204.66:45147)
0923 15:11:29.833780 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:29.833793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:29.833852 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:29.834038 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:29.834039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:29.834041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:29.834115 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.834117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.834872 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.834874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.835792 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.835795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.836394 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.836395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.836474 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.836477 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:29.836549 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 15:11:29.836592 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":139,"threads_started":1}
I20260923 15:11:29.837020 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.833672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:49039 (local address 127.21.204.67:39255)
0923 15:11:29.833821 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:29.833823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:29.833832 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:29.833908 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:29.833909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:29.833920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:29.833993 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:29.834341 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.834833 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.836133 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.836521 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.836616 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.836625 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:29.836883 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 15:11:29.836935 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260923 15:11:29.837011 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.833520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:39255 (local address 127.21.204.66:49039)
0923 15:11:29.833764 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:29.833777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:29.833839 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:29.833998 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:29.833999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:29.834001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:29.834249 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.834252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.834924 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.834925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.835995 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.835998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.836616 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.836618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.836773 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.836778 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:11:29.836837 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:11:29.836882 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":176,"threads_started":1}
I20260923 15:11:29.837234 24214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0f80a4aee24883bdc1367ec79c3f37" candidate_uuid: "47eeec232c7e4b1f94994abd14f63442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7974ed8621194b3fa1128f5e200db1ac" is_pre_election: true
I20260923 15:11:29.837356 24214 raft_consensus.cc:2454] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eeec232c7e4b1f94994abd14f63442 in term 0.
I20260923 15:11:29.837371 24494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0f80a4aee24883bdc1367ec79c3f37" candidate_uuid: "47eeec232c7e4b1f94994abd14f63442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9117b32e04c12b62b4364d5b523e4" is_pre_election: true
I20260923 15:11:29.837469 24494 raft_consensus.cc:2454] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47eeec232c7e4b1f94994abd14f63442 in term 0.
I20260923 15:11:29.837586 24288 leader_election.cc:302] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eeec232c7e4b1f94994abd14f63442, 7974ed8621194b3fa1128f5e200db1ac; no voters:
I20260923 15:11:29.837741 24657 raft_consensus.cc:2781] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:29.837788 24657 raft_consensus.cc:482] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:29.837808 24657 raft_consensus.cc:3037] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:29.838537 24657 raft_consensus.cc:504] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.838682 24657 leader_election.cc:288] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [CANDIDATE]: Term 1 election: Requested vote from peers 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621), 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255)
I20260923 15:11:29.838815 24214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0f80a4aee24883bdc1367ec79c3f37" candidate_uuid: "47eeec232c7e4b1f94994abd14f63442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7974ed8621194b3fa1128f5e200db1ac"
I20260923 15:11:29.838851 24494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd0f80a4aee24883bdc1367ec79c3f37" candidate_uuid: "47eeec232c7e4b1f94994abd14f63442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87f9117b32e04c12b62b4364d5b523e4"
I20260923 15:11:29.838894 24214 raft_consensus.cc:3037] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:29.838918 24494 raft_consensus.cc:3037] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:29.839733 24494 raft_consensus.cc:2454] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eeec232c7e4b1f94994abd14f63442 in term 1.
I20260923 15:11:29.839784 24214 raft_consensus.cc:2454] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47eeec232c7e4b1f94994abd14f63442 in term 1.
I20260923 15:11:29.839931 24287 leader_election.cc:302] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47eeec232c7e4b1f94994abd14f63442, 87f9117b32e04c12b62b4364d5b523e4; no voters:
I20260923 15:11:29.840044 24657 raft_consensus.cc:2781] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:29.840198 24657 raft_consensus.cc:686] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 1 LEADER]: Becoming Leader. State: Replica: 47eeec232c7e4b1f94994abd14f63442, State: Running, Role: LEADER
I20260923 15:11:29.840279 24657 consensus_queue.cc:237] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.841135 24057 catalog_manager.cc:5495] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "47eeec232c7e4b1f94994abd14f63442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:29.854923 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.852048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.21.204.66:46095)
0923 15:11:29.852299 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:29.852301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:29.852310 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:29.852395 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:29.852396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:29.852421 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:29.852519 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:29.852707 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.853426 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.854220 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.854412 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.854467 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.854645 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:29.854747 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:29.854804 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:29.854842 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 15:11:29.854858 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":130,"threads_started":1}
I20260923 15:11:29.860292 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.857753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:40021 (local address 127.21.204.67:39255)
0923 15:11:29.857824 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:29.857826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:29.857919 (+ 93us) server_negotiation.cc:407] Connection header received
0923 15:11:29.858022 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:29.858023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:29.858033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:29.858071 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:29.858200 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.858640 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.859508 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.859717 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.859782 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.860061 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:29.860091 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:29.860145 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:29.860220 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 15:11:29.860240 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 15:11:29.860533 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.857638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:39255 (local address 127.21.204.66:40021)
0923 15:11:29.857822 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:29.857833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:29.857942 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:29.858091 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:29.858091 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:29.858093 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:29.858140 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.858141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.858714 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.858714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.859415 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.859417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.859813 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.859814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.859939 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.859952 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:29.860162 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:29.860164 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:29.860201 (+ 37us) client_negotiation.cc:240] Negotiation successful
0923 15:11:29.860237 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260923 15:11:29.863426 24494 raft_consensus.cc:1261] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Refusing update from remote peer 47eeec232c7e4b1f94994abd14f63442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:29.863431 24214 raft_consensus.cc:1261] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Refusing update from remote peer 47eeec232c7e4b1f94994abd14f63442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:29.863734 24657 consensus_queue.cc:1035] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:29.863925 24662 consensus_queue.cc:1035] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:29.872047 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.869550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:44913 (local address 127.21.204.66:46095)
0923 15:11:29.869590 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:29.869592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:29.869789 (+ 197us) server_negotiation.cc:407] Connection header received
0923 15:11:29.869834 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:29.869835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:29.869844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:29.869940 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:29.870040 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.870488 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.871415 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.871659 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.871733 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.871867 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:29.871896 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:29.871933 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:29.871982 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 15:11:29.871998 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:11:29.872073 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:29.869516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46095 (local address 127.21.204.67:44913)
0923 15:11:29.869737 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:29.869746 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:29.869794 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:29.869945 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:29.869945 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:29.869947 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:29.870000 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.870001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.870549 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.870549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.871339 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:29.871342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:29.871723 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:29.871724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:29.871801 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:29.871805 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:29.871946 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:29.871948 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:29.871982 (+ 34us) client_negotiation.cc:240] Negotiation successful
0923 15:11:29.872018 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":142,"threads_started":1}
I20260923 15:11:29.878293 24057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:29.878402 24057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:29.879154 24057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Sending DeleteTablet for 3 replicas of tablet dd0f80a4aee24883bdc1367ec79c3f37
I20260923 15:11:29.879411 24474 tablet_service.cc:1514] Processing DeleteTablet for tablet dd0f80a4aee24883bdc1367ec79c3f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:29 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:59424
I20260923 15:11:29.879421 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet dd0f80a4aee24883bdc1367ec79c3f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:29 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:52940
I20260923 15:11:29.879520 24640 tablet_replica.cc:330] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: stopping tablet replica
I20260923 15:11:29.879643 24640 raft_consensus.cc:2229] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:29.879696 24640 raft_consensus.cc:2258] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:29.879667 24057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 15:11:29.879768 24642 tablet_replica.cc:330] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: stopping tablet replica
I20260923 15:11:29.879815 24642 raft_consensus.cc:2229] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 15:11:29.879814 24057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:29.879849 24642 raft_consensus.cc:2258] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:29.880292 24642 ts_tablet_manager.cc:1905] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:29.881175 24474 tablet_service.cc:1467] Processing CreateTablet for tablet 2756b26778c1463fbda05e6eb4b1d80a (DEFAULT_TABLE table=test-table [id=9a340a3ac6284f7c893e2f2420383f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:29.881294 24474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2756b26778c1463fbda05e6eb4b1d80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:29.881994 24194 tablet_service.cc:1467] Processing CreateTablet for tablet 2756b26778c1463fbda05e6eb4b1d80a (DEFAULT_TABLE table=test-table [id=9a340a3ac6284f7c893e2f2420383f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:29.882230 24640 ts_tablet_manager.cc:1905] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:29.882942 24651 tablet_bootstrap.cc:492] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: Bootstrap starting.
I20260923 15:11:29.883298 24651 tablet_bootstrap.cc:654] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:29.883862 24642 ts_tablet_manager.cc:1918] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:29.883873 24651 tablet_bootstrap.cc:492] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: No bootstrap required, opened a new log
I20260923 15:11:29.883914 24642 log.cc:1196] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/dd0f80a4aee24883bdc1367ec79c3f37
I20260923 15:11:29.883922 24651 ts_tablet_manager.cc:1397] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:29.883947 24640 ts_tablet_manager.cc:1918] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:29.883996 24640 log.cc:1196] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/dd0f80a4aee24883bdc1367ec79c3f37
I20260923 15:11:29.884063 24651 raft_consensus.cc:348] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.884135 24642 ts_tablet_manager.cc:1939] T dd0f80a4aee24883bdc1367ec79c3f37 P 7974ed8621194b3fa1128f5e200db1ac: Deleting consensus metadata
I20260923 15:11:29.884135 24651 raft_consensus.cc:374] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:29.884162 24651 raft_consensus.cc:729] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f9117b32e04c12b62b4364d5b523e4, State: Initialized, Role: FOLLOWER
I20260923 15:11:29.884214 24640 ts_tablet_manager.cc:1939] T dd0f80a4aee24883bdc1367ec79c3f37 P 87f9117b32e04c12b62b4364d5b523e4: Deleting consensus metadata
I20260923 15:11:29.884267 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2756b26778c1463fbda05e6eb4b1d80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:29.884481 24334 tablet_service.cc:1514] Processing DeleteTablet for tablet dd0f80a4aee24883bdc1367ec79c3f37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:29 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:48538
I20260923 15:11:29.884574 24641 tablet_replica.cc:330] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: stopping tablet replica
I20260923 15:11:29.884214 24651 consensus_queue.cc:260] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.884646 24641 raft_consensus.cc:2229] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:29.884711 24651 ts_tablet_manager.cc:1428] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:29.884735 24641 raft_consensus.cc:2258] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:29.884572 24333 tablet_service.cc:1467] Processing CreateTablet for tablet 2756b26778c1463fbda05e6eb4b1d80a (DEFAULT_TABLE table=test-table [id=9a340a3ac6284f7c893e2f2420383f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:29.885048 24333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2756b26778c1463fbda05e6eb4b1d80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:29.885432 24641 ts_tablet_manager.cc:1905] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:29.886122 24044 catalog_manager.cc:4841] TS 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621): tablet dd0f80a4aee24883bdc1367ec79c3f37 (table test-table [id=a3d286436a484353a189b6e35a6234d0]) successfully deleted
I20260923 15:11:29.887398 24641 ts_tablet_manager.cc:1918] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:29.887490 24641 log.cc:1196] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/dd0f80a4aee24883bdc1367ec79c3f37
I20260923 15:11:29.887703 24641 ts_tablet_manager.cc:1939] T dd0f80a4aee24883bdc1367ec79c3f37 P 47eeec232c7e4b1f94994abd14f63442: Deleting consensus metadata
I20260923 15:11:29.887737 24652 tablet_bootstrap.cc:492] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac: Bootstrap starting.
I20260923 15:11:29.887724 24653 tablet_bootstrap.cc:492] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: Bootstrap starting.
I20260923 15:11:29.888114 24653 tablet_bootstrap.cc:654] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:29.888187 24652 tablet_bootstrap.cc:654] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:29.888597 24044 catalog_manager.cc:4841] TS 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095): tablet dd0f80a4aee24883bdc1367ec79c3f37 (table test-table [id=a3d286436a484353a189b6e35a6234d0]) successfully deleted
I20260923 15:11:29.888746 24652 tablet_bootstrap.cc:492] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac: No bootstrap required, opened a new log
I20260923 15:11:29.888777 24652 ts_tablet_manager.cc:1397] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:29.888855 24652 raft_consensus.cc:348] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.888903 24652 raft_consensus.cc:374] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:29.888916 24652 raft_consensus.cc:729] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7974ed8621194b3fa1128f5e200db1ac, State: Initialized, Role: FOLLOWER
I20260923 15:11:29.888976 24652 consensus_queue.cc:260] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.889075 24652 ts_tablet_manager.cc:1428] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:29.889345 24043 catalog_manager.cc:4841] TS 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255): tablet dd0f80a4aee24883bdc1367ec79c3f37 (table test-table [id=a3d286436a484353a189b6e35a6234d0]) successfully deleted
I20260923 15:11:29.889711 24653 tablet_bootstrap.cc:492] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: No bootstrap required, opened a new log
I20260923 15:11:29.889761 24653 ts_tablet_manager.cc:1397] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:29.890076 24653 raft_consensus.cc:348] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.890144 24653 raft_consensus.cc:374] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:29.890166 24653 raft_consensus.cc:729] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eeec232c7e4b1f94994abd14f63442, State: Initialized, Role: FOLLOWER
I20260923 15:11:29.890225 24653 consensus_queue.cc:260] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:29.890448 24653 ts_tablet_manager.cc:1428] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 15:11:30.169330 24570 raft_consensus.cc:482] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:30.169430 24570 raft_consensus.cc:504] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:30.169593 24570 leader_election.cc:288] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621), 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095)
I20260923 15:11:30.169871 24214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2756b26778c1463fbda05e6eb4b1d80a" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7974ed8621194b3fa1128f5e200db1ac" is_pre_election: true
I20260923 15:11:30.169932 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2756b26778c1463fbda05e6eb4b1d80a" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442" is_pre_election: true
I20260923 15:11:30.170006 24214 raft_consensus.cc:2454] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 0.
I20260923 15:11:30.170015 24354 raft_consensus.cc:2454] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 0.
I20260923 15:11:30.170167 24428 leader_election.cc:302] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7974ed8621194b3fa1128f5e200db1ac, 87f9117b32e04c12b62b4364d5b523e4; no voters:
I20260923 15:11:30.170291 24570 raft_consensus.cc:2781] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:30.170374 24570 raft_consensus.cc:482] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:30.170409 24570 raft_consensus.cc:3037] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:30.171111 24570 raft_consensus.cc:504] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:30.171258 24570 leader_election.cc:288] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 election: Requested vote from peers 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621), 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095)
I20260923 15:11:30.171459 24214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2756b26778c1463fbda05e6eb4b1d80a" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7974ed8621194b3fa1128f5e200db1ac"
I20260923 15:11:30.171491 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2756b26778c1463fbda05e6eb4b1d80a" candidate_uuid: "87f9117b32e04c12b62b4364d5b523e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eeec232c7e4b1f94994abd14f63442"
I20260923 15:11:30.171538 24214 raft_consensus.cc:3037] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:30.171568 24354 raft_consensus.cc:3037] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:30.172226 24354 raft_consensus.cc:2454] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 1.
I20260923 15:11:30.172240 24214 raft_consensus.cc:2454] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f9117b32e04c12b62b4364d5b523e4 in term 1.
I20260923 15:11:30.172418 24428 leader_election.cc:302] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7974ed8621194b3fa1128f5e200db1ac, 87f9117b32e04c12b62b4364d5b523e4; no voters:
I20260923 15:11:30.172528 24570 raft_consensus.cc:2781] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:30.172598 24570 raft_consensus.cc:686] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 1 LEADER]: Becoming Leader. State: Replica: 87f9117b32e04c12b62b4364d5b523e4, State: Running, Role: LEADER
I20260923 15:11:30.172650 24570 consensus_queue.cc:237] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } }
I20260923 15:11:30.173300 24057 catalog_manager.cc:5495] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67). New cstate: current_term: 1 leader_uuid: "87f9117b32e04c12b62b4364d5b523e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87f9117b32e04c12b62b4364d5b523e4" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 39255 } health_report { overall_health: HEALTHY } } }
I20260923 15:11:30.239972 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:30.238056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.21.204.67:39255)
0923 15:11:30.238129 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:30.238130 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:30.238137 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:30.238156 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:30.238157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:30.238168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:30.238217 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:30.238326 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:30.238783 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:30.239439 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:30.239648 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:30.239700 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:30.239813 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:30.239836 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:30.239866 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:30.239900 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 15:11:30.239916 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 15:11:30.244307 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:30.242376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:56721 (local address 127.21.204.67:39255)
0923 15:11:30.242408 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:30.242410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:30.242416 (+ 6us) server_negotiation.cc:407] Connection header received
0923 15:11:30.242432 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:30.242433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:30.242439 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:30.242484 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:30.242592 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:30.242996 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:30.243761 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:30.243929 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:30.243981 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:30.244091 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:30.244125 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:30.244162 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:30.244224 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 15:11:30.244241 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":16}
I20260923 15:11:30.244354 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:30.242172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:39255 (local address 127.21.204.67:56721)
0923 15:11:30.242227 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:30.242238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:30.242295 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:30.242502 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:30.242503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:30.242504 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:30.242534 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:30.242535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:30.243068 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:30.243068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:30.243692 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:30.243695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:30.243994 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:30.243995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:30.244032 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:30.244037 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:30.244186 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:30.244188 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:30.244223 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 15:11:30.244260 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260923 15:11:30.246366 24354 raft_consensus.cc:1261] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Refusing update from remote peer 87f9117b32e04c12b62b4364d5b523e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:30.246405 24214 raft_consensus.cc:1261] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac [term 1 FOLLOWER]: Refusing update from remote peer 87f9117b32e04c12b62b4364d5b523e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:30.246656 24570 consensus_queue.cc:1035] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eeec232c7e4b1f94994abd14f63442" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:30.246763 24621 consensus_queue.cc:1035] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7974ed8621194b3fa1128f5e200db1ac" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 15:11:30.251351 24619 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 15:11:31.772583 24055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:31.772722 24055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:31.773591 24055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cccebd1fd124d85993a808f5bfe141f: Sending DeleteTablet for 3 replicas of tablet 2756b26778c1463fbda05e6eb4b1d80a
I20260923 15:11:31.773929 24333 tablet_service.cc:1514] Processing DeleteTablet for tablet 2756b26778c1463fbda05e6eb4b1d80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:31 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:48538
I20260923 15:11:31.774031 24194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2756b26778c1463fbda05e6eb4b1d80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:31 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:52940
I20260923 15:11:31.774026 24474 tablet_service.cc:1514] Processing DeleteTablet for tablet 2756b26778c1463fbda05e6eb4b1d80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:31 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:59424
I20260923 15:11:31.774178 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 24129
I20260923 15:11:31.774205 24703 tablet_replica.cc:330] T 2756b26778c1463fbda05e6eb4b1d80a P 7974ed8621194b3fa1128f5e200db1ac: stopping tablet replica
I20260923 15:11:31.774797 24702 tablet_replica.cc:330] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: stopping tablet replica
I20260923 15:11:31.774868 24702 raft_consensus.cc:2229] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:31.774961 24702 raft_consensus.cc:2258] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:31.775100 24704 tablet_replica.cc:330] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: stopping tablet replica
I20260923 15:11:31.775205 24704 raft_consensus.cc:2229] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:31.775310 24704 raft_consensus.cc:2258] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:31.775302 24702 ts_tablet_manager.cc:1905] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:31.775525 24704 ts_tablet_manager.cc:1905] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:31.777073 24704 ts_tablet_manager.cc:1918] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 15:11:31.777153 24704 log.cc:1196] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/2756b26778c1463fbda05e6eb4b1d80a
I20260923 15:11:31.777307 24702 ts_tablet_manager.cc:1918] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 15:11:31.777379 24702 log.cc:1196] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/2756b26778c1463fbda05e6eb4b1d80a
I20260923 15:11:31.777480 24704 ts_tablet_manager.cc:1939] T 2756b26778c1463fbda05e6eb4b1d80a P 87f9117b32e04c12b62b4364d5b523e4: Deleting consensus metadata
I20260923 15:11:31.777657 24702 ts_tablet_manager.cc:1939] T 2756b26778c1463fbda05e6eb4b1d80a P 47eeec232c7e4b1f94994abd14f63442: Deleting consensus metadata
I20260923 15:11:31.777828 24043 catalog_manager.cc:4841] TS 87f9117b32e04c12b62b4364d5b523e4 (127.21.204.67:39255): tablet 2756b26778c1463fbda05e6eb4b1d80a (table test-table [id=9a340a3ac6284f7c893e2f2420383f3b]) successfully deleted
I20260923 15:11:31.778052 24044 catalog_manager.cc:4841] TS 47eeec232c7e4b1f94994abd14f63442 (127.21.204.66:46095): tablet 2756b26778c1463fbda05e6eb4b1d80a (table test-table [id=9a340a3ac6284f7c893e2f2420383f3b]) successfully deleted
W20260923 15:11:31.782804 24044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:44621 (error 108)
I20260923 15:11:31.783020 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 24269
I20260923 15:11:31.783839 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:31.783226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44621 (local address 127.0.0.1:37416)
0923 15:11:31.783734 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:31.783786 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":71,"threads_started":1}
W20260923 15:11:31.783939 24044 catalog_manager.cc:4584] TS 7974ed8621194b3fa1128f5e200db1ac (127.21.204.65:44621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2756b26778c1463fbda05e6eb4b1d80a: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44621: connect: Connection refused (error 111)
I20260923 15:11:31.789706 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 24409
I20260923 15:11:31.796392 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 24025
2026-09-23T15:11:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6299 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:31 dist-test-slave-t5lt krb5kdc[24709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:31 dist-test-slave-t5lt krb5kdc[24709](info): set up 2 sockets
Sep 23 15:11:31 dist-test-slave-t5lt krb5kdc[24709](info): commencing operation
krb5kdc: starting...
W20260923 15:11:32.851105 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:32 dist-test-slave-t5lt krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:32.977607 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:46239
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:33.059062 24722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:33.059306 24722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:33.059345 24722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:33.060473 24722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:33.060541 24722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:33.060567 24722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:33.060595 24722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:33.060616 24722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:33.062168 24722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:46239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:33.062441 24722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:33.062668 24722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:33.068209 24728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.068292 24722 server_base.cc:1034] running on GCE node
W20260923 15:11:33.068239 24727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:33.068194 24730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.069279 24722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:33.069533 24722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:33.070673 24722 hybrid_clock.cc:648] HybridClock initialized: now 1790176293070655 us; error 40 us; skew 500 ppm
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:33.073213 24722 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:33.073663 24722 webserver.cc:460] Webserver started at http://127.21.204.126:44061/ using document root <none> and password file <none>
I20260923 15:11:33.073863 24722 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:33.073964 24722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:33.074059 24722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:33.075045 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "9e83550c9af6489d9f722bdcabdb967a"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "c5de3f1574c406874cae6c33b6d7991c"
server_key_iv: "8798451eb3cdf770920ca84fd0ffa63b"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.075346 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "9e83550c9af6489d9f722bdcabdb967a"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "c5de3f1574c406874cae6c33b6d7991c"
server_key_iv: "8798451eb3cdf770920ca84fd0ffa63b"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.076565 24722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:11:33.077253 24737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.077423 24722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:33.077510 24722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "9e83550c9af6489d9f722bdcabdb967a"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "c5de3f1574c406874cae6c33b6d7991c"
server_key_iv: "8798451eb3cdf770920ca84fd0ffa63b"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.077582 24722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:33.085067 24722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:33.087339 24722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:33.087496 24722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:33.091372 24722 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:46239
I20260923 15:11:33.091434 24789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:46239 every 8 connection(s)
I20260923 15:11:33.091859 24722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:33.092394 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 24722
I20260923 15:11:33.092499 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:33.092716 22321 external_mini_cluster.cc:1428] Setting key eff4153f5eee2cad668446199cfdb336
I20260923 15:11:33.092847 24790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:33.096179 24790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a: Bootstrap starting.
I20260923 15:11:33.096875 24790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:33.097151 24790 log.cc:824] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:33.097817 24790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a: No bootstrap required, opened a new log
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:33.099858 24790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e83550c9af6489d9f722bdcabdb967a" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46239 } }
I20260923 15:11:33.100030 24790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:33.100101 24790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e83550c9af6489d9f722bdcabdb967a, State: Initialized, Role: FOLLOWER
I20260923 15:11:33.100239 24790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e83550c9af6489d9f722bdcabdb967a" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46239 } }
I20260923 15:11:33.100325 24790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:33.100392 24790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:33.100463 24790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:33.101189 24790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e83550c9af6489d9f722bdcabdb967a" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46239 } }
I20260923 15:11:33.101298 24790 leader_election.cc:302] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e83550c9af6489d9f722bdcabdb967a; no voters:
I20260923 15:11:33.101493 24790 leader_election.cc:288] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:33.101656 24795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:33.101752 24790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:33.101814 24795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [term 1 LEADER]: Becoming Leader. State: Replica: 9e83550c9af6489d9f722bdcabdb967a, State: Running, Role: LEADER
I20260923 15:11:33.101945 24795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e83550c9af6489d9f722bdcabdb967a" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46239 } }
I20260923 15:11:33.102301 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e83550c9af6489d9f722bdcabdb967a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e83550c9af6489d9f722bdcabdb967a" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46239 } } }
I20260923 15:11:33.102404 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:33.102632 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e83550c9af6489d9f722bdcabdb967a. Latest consensus state: current_term: 1 leader_uuid: "9e83550c9af6489d9f722bdcabdb967a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e83550c9af6489d9f722bdcabdb967a" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46239 } } }
I20260923 15:11:33.102692 24804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:33.102759 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:33.103287 24804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:33.104713 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:33.093313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.21.204.126:46239)
0923 15:11:33.093488 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:33.093496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:33.093519 (+ 23us) server_negotiation.cc:407] Connection header received
0923 15:11:33.094007 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:33.094014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:33.094061 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:33.094186 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:33.094991 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.095849 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.096905 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.097418 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.097565 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:33.100371 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:33.100376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:33.100376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:33.100388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:33.102585 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.103059 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.103063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.103063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.103180 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.103419 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.103420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.103420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.103742 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:33.104294 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:33.104378 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 15:11:33.104517 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260923 15:11:33.105377 24804 catalog_manager.cc:1347] Generated new cluster ID: dba62c7e803f44acbfa2538afb8fa606
I20260923 15:11:33.105437 24804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:33.128361 24804 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:33.129113 24804 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:33.140363 24804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e83550c9af6489d9f722bdcabdb967a: Generated new TSK 0
I20260923 15:11:33.140574 24804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:33.170218 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46239
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:33.254894 24816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:33.255103 24816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:33.255128 24816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:33.256325 24816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:33.256367 24816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:33.256395 24816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:33.257843 24816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:33.258149 24816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:33.258387 24816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:33.263569 24821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.263841 24816 server_base.cc:1034] running on GCE node
W20260923 15:11:33.263630 24824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:33.263569 24822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.264230 24816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:33.264484 24816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:33.265753 24816 hybrid_clock.cc:648] HybridClock initialized: now 1790176293265733 us; error 31 us; skew 500 ppm
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:33.268381 24816 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:33.268855 24816 webserver.cc:460] Webserver started at http://127.21.204.65:42143/ using document root <none> and password file <none>
I20260923 15:11:33.269101 24816 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:33.269148 24816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:33.269320 24816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:33.270506 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "79132004beda4481bd924cd77aa8a40c"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "498436311d2303193fbcbbc73a60a918"
server_key_iv: "33b5f4aab59256ddb0fc2f695feec2ed"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.270881 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "79132004beda4481bd924cd77aa8a40c"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "498436311d2303193fbcbbc73a60a918"
server_key_iv: "33b5f4aab59256ddb0fc2f695feec2ed"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.272289 24816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:11:33.273310 24831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.273579 24816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:33.273715 24816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "79132004beda4481bd924cd77aa8a40c"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "498436311d2303193fbcbbc73a60a918"
server_key_iv: "33b5f4aab59256ddb0fc2f695feec2ed"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.273811 24816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:33.282986 24816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:33.285243 24816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:33.285420 24816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:33.285742 24816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:33.286141 24816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:33.286203 24816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.286248 24816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:33.286274 24816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.292819 24816 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:34841
I20260923 15:11:33.292878 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:34841 every 8 connection(s)
I20260923 15:11:33.293244 24816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:33.295275 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 24816
I20260923 15:11:33.295367 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:33.295692 22321 external_mini_cluster.cc:1428] Setting key 63ae1c1b37092933159691ed104a8332
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:11:33.304312 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:33.294224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:44361 (local address 127.21.204.126:46239)
0923 15:11:33.294279 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:33.294282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:33.294531 (+ 249us) server_negotiation.cc:407] Connection header received
0923 15:11:33.295091 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:33.295093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:33.295109 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:33.295200 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:33.296456 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.297204 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.298371 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.298614 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.298700 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:33.301159 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:33.301162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:33.301162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:33.301175 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:33.302821 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.303260 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.303261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.303262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.303324 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.303628 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.303629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.303629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.303751 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:33.303829 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:33.304128 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 15:11:33.304217 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:33.304535 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:33.294211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:46239 (local address 127.21.204.65:44361)
0923 15:11:33.294403 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:33.294441 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:33.294926 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:33.295305 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:33.295309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:33.296070 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:33.296342 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:33.296350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.297362 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.297363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:33.298273 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:33.298276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.298710 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.298711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:33.298846 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:33.299430 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:33.299440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:33.301028 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:33.302917 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:33.302926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:33.302928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:33.303164 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:33.303397 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:33.303398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:33.303398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:33.303565 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:33.303833 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:33.303841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:33.303988 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 15:11:33.304101 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 15:11:33.304268 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260923 15:11:33.305238 24945 heartbeater.cc:347] Connected to a master server at 127.21.204.126:46239
I20260923 15:11:33.305351 24945 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:33.305546 24945 heartbeater.cc:511] Master 127.21.204.126:46239 requested a full tablet report, sending...
I20260923 15:11:33.306167 24754 ts_manager.cc:194] Registered new tserver with Master: 79132004beda4481bd924cd77aa8a40c (127.21.204.65:34841)
I20260923 15:11:33.306931 24754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:44361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:33.326458 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46239
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:33.410647 24950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:33.410900 24950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:33.410939 24950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:33.412114 24950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:33.412194 24950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:33.412240 24950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:33.413681 24950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:33.413988 24950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:33.414243 24950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:33.419440 24955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:33.419451 24958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.419446 24950 server_base.cc:1034] running on GCE node
W20260923 15:11:33.419451 24956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.419998 24950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:33.420277 24950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:33.421468 24950 hybrid_clock.cc:648] HybridClock initialized: now 1790176293421415 us; error 73 us; skew 500 ppm
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:33.424166 24950 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:33.424636 24950 webserver.cc:460] Webserver started at http://127.21.204.66:44579/ using document root <none> and password file <none>
I20260923 15:11:33.424886 24950 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:33.424954 24950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:33.425113 24950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:33.426244 24950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "1446ac9b49a442f09b9163c90ad6bc4a"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "bb5a8e5a2aee3d3385d9bf181ed62213"
server_key_iv: "53b5cee6a10c731932f98ccb5887c44d"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.426606 24950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "1446ac9b49a442f09b9163c90ad6bc4a"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "bb5a8e5a2aee3d3385d9bf181ed62213"
server_key_iv: "53b5cee6a10c731932f98ccb5887c44d"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.427989 24950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:33.428828 24965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.429039 24950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:33.429147 24950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "1446ac9b49a442f09b9163c90ad6bc4a"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "bb5a8e5a2aee3d3385d9bf181ed62213"
server_key_iv: "53b5cee6a10c731932f98ccb5887c44d"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.429255 24950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:33.441424 24950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:33.443596 24950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:33.443782 24950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:33.444025 24950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:33.444358 24950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:33.444416 24950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.444458 24950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:33.444485 24950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.450423 24950 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:35479
I20260923 15:11:33.450491 25078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:35479 every 8 connection(s)
I20260923 15:11:33.450838 24950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:33.451711 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 24950
I20260923 15:11:33.451819 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:33.452009 22321 external_mini_cluster.cc:1428] Setting key 9170a47000c41719aff3953234fc0839
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
I20260923 15:11:33.460793 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:33.451657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:51307 (local address 127.21.204.126:46239)
0923 15:11:33.451731 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:33.451733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:33.451863 (+ 130us) server_negotiation.cc:407] Connection header received
0923 15:11:33.452516 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:33.452517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:33.452530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:33.452580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:33.453567 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.454151 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.455156 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.455355 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.455429 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:33.457844 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:33.457846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:33.457846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:33.457854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:33.459308 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.459775 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.459777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.459777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.459854 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.460112 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.460113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.460113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.460232 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:33.460310 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:33.460618 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 15:11:33.460702 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 15:11:33.460981 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:33.451563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:46239 (local address 127.21.204.66:51307)
0923 15:11:33.451743 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:33.451784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:33.452405 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:33.452658 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:33.452662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:33.453273 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:33.453463 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:33.453468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.454246 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.454246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:33.455050 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:33.455053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.455447 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.455448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:33.455573 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:33.456107 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:33.456116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:33.457742 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:33.459414 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:33.459421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:33.459423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:33.459680 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:33.459924 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:33.459925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:33.459925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:33.460037 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:33.460301 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:33.460308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:33.460476 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 15:11:33.460589 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 15:11:33.460746 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260923 15:11:33.461536 25079 heartbeater.cc:347] Connected to a master server at 127.21.204.126:46239
I20260923 15:11:33.461685 25079 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:33.461871 25079 heartbeater.cc:511] Master 127.21.204.126:46239 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
I20260923 15:11:33.462371 24754 ts_manager.cc:194] Registered new tserver with Master: 1446ac9b49a442f09b9163c90ad6bc4a (127.21.204.66:35479)
I20260923 15:11:33.462766 24754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:51307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:33.477327 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46239
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:33.557642 25084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:33.557832 25084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:33.557857 25084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:33.559037 25084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:33.559083 25084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:33.559110 25084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:33.560509 25084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:33.560752 25084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:33.560990 25084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:33.566026 25090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:33.566009 25089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.566109 25084 server_base.cc:1034] running on GCE node
W20260923 15:11:33.566010 25092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:33.566434 25084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:33.566646 25084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:33.567849 25084 hybrid_clock.cc:648] HybridClock initialized: now 1790176293567836 us; error 30 us; skew 500 ppm
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:33.570437 25084 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:33.570897 25084 webserver.cc:460] Webserver started at http://127.21.204.67:44005/ using document root <none> and password file <none>
I20260923 15:11:33.571133 25084 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:33.571205 25084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:33.571357 25084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:33.572621 25084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "bf55369e55ce4ac686a3b37f16133eb6"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "3ea291010ef11fb626f232cbe3758c86"
server_key_iv: "f4bea724c8262a8a2caafef148c967f1"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.572983 25084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "bf55369e55ce4ac686a3b37f16133eb6"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "3ea291010ef11fb626f232cbe3758c86"
server_key_iv: "f4bea724c8262a8a2caafef148c967f1"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.574401 25084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:33.575145 25099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.575327 25084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:33.575431 25084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "bf55369e55ce4ac686a3b37f16133eb6"
format_stamp: "Formatted at 2026-09-23 15:11:33 on dist-test-slave-t5lt"
server_key: "3ea291010ef11fb626f232cbe3758c86"
server_key_iv: "f4bea724c8262a8a2caafef148c967f1"
server_key_version: "encryptionkey@0"
I20260923 15:11:33.575506 25084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:33.585515 25084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:33.587778 25084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:33.587939 25084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:33.588179 25084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:33.588508 25084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:33.588567 25084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.588641 25084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:33.588671 25084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:33.594950 25084 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:40231
I20260923 15:11:33.594992 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:40231 every 8 connection(s)
I20260923 15:11:33.595309 25084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[24709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176293, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:33.602594 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25084
I20260923 15:11:33.602715 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:33.602965 22321 external_mini_cluster.cc:1428] Setting key 1488bb2b24db359c0cd818e1c95fa6ac
I20260923 15:11:33.605172 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:33.596085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:46947 (local address 127.21.204.126:46239)
0923 15:11:33.596170 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:33.596173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:33.596364 (+ 191us) server_negotiation.cc:407] Connection header received
0923 15:11:33.596935 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:33.596937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:33.596950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:33.597193 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:33.598253 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.598757 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.599749 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.599943 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.600021 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:33.602314 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:33.602316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:33.602316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:33.602324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:33.603990 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.604365 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.604367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.604367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.604432 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:33.604651 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:33.604652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:33.604653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:33.604767 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:33.604823 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:33.605037 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 15:11:33.605098 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 15:11:33.605363 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:33.596032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:46239 (local address 127.21.204.67:46947)
0923 15:11:33.596219 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:33.596257 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:33.596815 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:33.597249 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:33.597253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:33.597932 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:33.598149 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:33.598154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.598857 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.598859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:33.599642 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:33.599644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:33.600029 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:33.600030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:33.600153 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:33.600643 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:33.600651 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:33.602223 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:33.604047 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:33.604053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:33.604055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:33.604284 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:33.604500 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:33.604501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:33.604501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:33.604596 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:33.604816 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:33.604819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:33.604931 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 15:11:33.605009 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 15:11:33.605125 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260923 15:11:33.605804 25213 heartbeater.cc:347] Connected to a master server at 127.21.204.126:46239
I20260923 15:11:33.605931 25213 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:33.606115 25213 heartbeater.cc:511] Master 127.21.204.126:46239 requested a full tablet report, sending...
I20260923 15:11:33.606494 24754 ts_manager.cc:194] Registered new tserver with Master: bf55369e55ce4ac686a3b37f16133eb6 (127.21.204.67:40231)
I20260923 15:11:33.606881 24754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:46947
I20260923 15:11:33.614651 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 15:11:33.626595 24793 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:33.625598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.21.204.126:46239)
0923 15:11:33.625682 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:33.625685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:33.625804 (+ 119us) server_negotiation.cc:407] Connection header received
0923 15:11:33.625903 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:33.625904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:33.625916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:33.625987 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:33.626507 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49012: BlockingRecv error: recv got EOF from 127.0.0.1:49012 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:11:33.627913 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 24816
I20260923 15:11:33.633136 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 24950
I20260923 15:11:33.637562 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25084
I20260923 15:11:33.641974 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 24722
2026-09-23T15:11:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1844 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[25225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[25225](info): set up 2 sockets
Sep 23 15:11:33 dist-test-slave-t5lt krb5kdc[25225](info): commencing operation
krb5kdc: starting...
W20260923 15:11:34.691270 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:34 dist-test-slave-t5lt krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:34.817970 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34033
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:34.919488 25238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:34.919705 25238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:34.919730 25238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:34.920924 25238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:34.920966 25238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:34.920980 25238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:34.920993 25238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:34.921047 25238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:34.922758 25238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:34.923025 25238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:34.923281 25238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:34.928447 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:34.928464 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:34.928483 25238 server_base.cc:1034] running on GCE node
W20260923 15:11:34.928464 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:34.928905 25238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:34.929194 25238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:34.930390 25238 hybrid_clock.cc:648] HybridClock initialized: now 1790176294930375 us; error 29 us; skew 500 ppm
Sep 23 15:11:34 dist-test-slave-t5lt krb5kdc[25225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176294, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:34.932963 25238 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:34.933462 25238 webserver.cc:460] Webserver started at http://127.21.204.126:40957/ using document root <none> and password file <none>
I20260923 15:11:34.933696 25238 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:34.933774 25238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:34.933961 25238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:34.935060 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "7d7179cd71d94f738c9421dbbd2686fd"
format_stamp: "Formatted at 2026-09-23 15:11:34 on dist-test-slave-t5lt"
server_key: "273b6410dd03f6f6c27eb4400ac3f0e9"
server_key_iv: "255997734efaaa74df1f116fdc08f5cd"
server_key_version: "encryptionkey@0"
I20260923 15:11:34.935431 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "7d7179cd71d94f738c9421dbbd2686fd"
format_stamp: "Formatted at 2026-09-23 15:11:34 on dist-test-slave-t5lt"
server_key: "273b6410dd03f6f6c27eb4400ac3f0e9"
server_key_iv: "255997734efaaa74df1f116fdc08f5cd"
server_key_version: "encryptionkey@0"
I20260923 15:11:34.936919 25238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:34.937935 25253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:34.938228 25238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:34.938345 25238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "7d7179cd71d94f738c9421dbbd2686fd"
format_stamp: "Formatted at 2026-09-23 15:11:34 on dist-test-slave-t5lt"
server_key: "273b6410dd03f6f6c27eb4400ac3f0e9"
server_key_iv: "255997734efaaa74df1f116fdc08f5cd"
server_key_version: "encryptionkey@0"
I20260923 15:11:34.938442 25238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:34.963003 25238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:34.965337 25238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:34.965539 25238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:34.970386 25238 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34033
I20260923 15:11:34.970463 25305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34033 every 8 connection(s)
I20260923 15:11:34.971002 25238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:34.971913 25306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:34.974586 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd: Bootstrap starting.
I20260923 15:11:34.975294 25306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:34.975399 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25238
I20260923 15:11:34.975515 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:34.975699 25306 log.cc:824] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:34.975739 22321 external_mini_cluster.cc:1428] Setting key 0d114e3af729dcdce8549e6a20e9dac3
I20260923 15:11:34.976621 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd: No bootstrap required, opened a new log
I20260923 15:11:34.979452 25306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7179cd71d94f738c9421dbbd2686fd" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34033 } }
I20260923 15:11:34.979666 25306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:34.979709 25306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7179cd71d94f738c9421dbbd2686fd, State: Initialized, Role: FOLLOWER
I20260923 15:11:34.979848 25306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7179cd71d94f738c9421dbbd2686fd" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34033 } }
I20260923 15:11:34.979918 25306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:34.979949 25306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:34.980005 25306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:34.981087 25306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7179cd71d94f738c9421dbbd2686fd" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34033 } }
I20260923 15:11:34.981213 25306 leader_election.cc:302] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d7179cd71d94f738c9421dbbd2686fd; no voters:
I20260923 15:11:34.981431 25306 leader_election.cc:288] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:34.981472 25311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:34.981670 25311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [term 1 LEADER]: Becoming Leader. State: Replica: 7d7179cd71d94f738c9421dbbd2686fd, State: Running, Role: LEADER
I20260923 15:11:34.981737 25306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:34.981859 25311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7179cd71d94f738c9421dbbd2686fd" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34033 } }
I20260923 15:11:34.982288 25312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d7179cd71d94f738c9421dbbd2686fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7179cd71d94f738c9421dbbd2686fd" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34033 } } }
I20260923 15:11:34.982367 25312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:34.982540 25313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d7179cd71d94f738c9421dbbd2686fd. Latest consensus state: current_term: 1 leader_uuid: "7d7179cd71d94f738c9421dbbd2686fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7179cd71d94f738c9421dbbd2686fd" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34033 } } }
I20260923 15:11:34.982596 25313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:34.983203 25318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 23 15:11:34 dist-test-slave-t5lt krb5kdc[25225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:34.984000 25318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:34.987320 25318 catalog_manager.cc:1347] Generated new cluster ID: b38342ce343b41febc87521e314634fd
I20260923 15:11:34.987428 25318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:34.988582 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:34.976299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.21.204.126:34033)
0923 15:11:34.976497 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:34.976505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:34.976531 (+ 26us) server_negotiation.cc:407] Connection header received
0923 15:11:34.977043 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:34.977051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:34.977105 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:34.977232 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:34.978187 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:34.979107 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:34.980292 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:34.980755 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:34.980846 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:34.984145 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:34.984149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:34.984150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:34.984164 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:34.986408 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:34.987095 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:34.987100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:34.987100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:34.987230 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:34.987451 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:34.987452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:34.987452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:34.987729 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:34.987929 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:34.988004 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 15:11:34.988300 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260923 15:11:34.991474 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:11:34.996842 25318 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:34.997429 25318 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:35.012769 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:35.007986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.21.204.126:34033)
0923 15:11:35.008036 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:35.008038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:35.008166 (+ 128us) server_negotiation.cc:407] Connection header received
0923 15:11:35.008244 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:35.008245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:35.008257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:35.008331 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:35.008814 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:35.009618 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:35.010778 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:35.011101 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:35.011159 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:35.012120 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:35.012121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:35.012122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:35.012361 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:35.012383 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:35.012390 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:35.012458 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:35.012545 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 15:11:35.012575 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:35.012849 25318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d7179cd71d94f738c9421dbbd2686fd: Generated new TSK 0
I20260923 15:11:35.013054 25318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:11:35.020998 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25238
2026-09-23T15:11:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1378 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:35 dist-test-slave-t5lt krb5kdc[25339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:35 dist-test-slave-t5lt krb5kdc[25339](info): set up 2 sockets
Sep 23 15:11:35 dist-test-slave-t5lt krb5kdc[25339](info): commencing operation
krb5kdc: starting...
W20260923 15:11:36.087008 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:36.225409 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40079
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:40079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:36.310319 25352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:36.310549 25352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:36.310575 25352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:36.311861 25352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:36.311910 25352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:36.311926 25352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:36.311940 25352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:36.311997 25352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:36.313624 25352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:40079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:36.313915 25352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:36.314193 25352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:36.319868 25357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:36.319875 25358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:36.319846 25360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:36.320152 25352 server_base.cc:1034] running on GCE node
I20260923 15:11:36.320482 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:36.320794 25352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:36.321945 25352 hybrid_clock.cc:648] HybridClock initialized: now 1790176296321930 us; error 33 us; skew 500 ppm
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:36.324527 25352 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:36.325062 25352 webserver.cc:460] Webserver started at http://127.21.204.126:40505/ using document root <none> and password file <none>
I20260923 15:11:36.325306 25352 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:36.325378 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:36.325531 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:36.326696 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "e3beef4bcb9c4d7a890744f220bbff00"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "f445eed06a6ccfc66fb2970f27b34487"
server_key_iv: "d5a8781118efefac0268923cb22be1ef"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.327078 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "e3beef4bcb9c4d7a890744f220bbff00"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "f445eed06a6ccfc66fb2970f27b34487"
server_key_iv: "d5a8781118efefac0268923cb22be1ef"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.328902 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 15:11:36.329823 25367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.330122 25352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:11:36.330252 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "e3beef4bcb9c4d7a890744f220bbff00"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "f445eed06a6ccfc66fb2970f27b34487"
server_key_iv: "d5a8781118efefac0268923cb22be1ef"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.330336 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:36.356567 25352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:36.358958 25352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:36.359124 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:36.363691 25352 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:40079
I20260923 15:11:36.363725 25419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:40079 every 8 connection(s)
I20260923 15:11:36.364109 25352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:36.365130 25420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:36.367655 25420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00: Bootstrap starting.
I20260923 15:11:36.368291 25420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:36.368592 25420 log.cc:824] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:36.369658 25420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00: No bootstrap required, opened a new log
I20260923 15:11:36.370723 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25352
I20260923 15:11:36.370847 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:36.371023 22321 external_mini_cluster.cc:1428] Setting key de6fc4fa4046e5ec4598bd250d996ead
I20260923 15:11:36.371821 25420 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3beef4bcb9c4d7a890744f220bbff00" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40079 } }
I20260923 15:11:36.372016 25420 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:36.372069 25420 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3beef4bcb9c4d7a890744f220bbff00, State: Initialized, Role: FOLLOWER
I20260923 15:11:36.372236 25420 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3beef4bcb9c4d7a890744f220bbff00" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40079 } }
I20260923 15:11:36.372335 25420 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:36.372397 25420 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:36.372452 25420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:36.373421 25420 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3beef4bcb9c4d7a890744f220bbff00" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40079 } }
I20260923 15:11:36.373553 25420 leader_election.cc:302] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3beef4bcb9c4d7a890744f220bbff00; no voters:
I20260923 15:11:36.373739 25420 leader_election.cc:288] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:36.373780 25425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:36.373919 25425 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [term 1 LEADER]: Becoming Leader. State: Replica: e3beef4bcb9c4d7a890744f220bbff00, State: Running, Role: LEADER
I20260923 15:11:36.374037 25420 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:36.374073 25425 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3beef4bcb9c4d7a890744f220bbff00" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40079 } }
I20260923 15:11:36.374517 25425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3beef4bcb9c4d7a890744f220bbff00. Latest consensus state: current_term: 1 leader_uuid: "e3beef4bcb9c4d7a890744f220bbff00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3beef4bcb9c4d7a890744f220bbff00" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40079 } } }
I20260923 15:11:36.374611 25426 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3beef4bcb9c4d7a890744f220bbff00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3beef4bcb9c4d7a890744f220bbff00" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40079 } } }
I20260923 15:11:36.374655 25425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:36.374782 25426 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:36.375030 25434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:36.375646 25434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:36.377416 25434 catalog_manager.cc:1347] Generated new cluster ID: ef6f67f4905d479cbb6c50eeff061d6f
I20260923 15:11:36.377483 25434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:36.382514 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.371726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.21.204.126:40079)
0923 15:11:36.371928 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:36.371936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:36.371962 (+ 26us) server_negotiation.cc:407] Connection header received
0923 15:11:36.372090 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:36.372097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:36.372156 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:36.372303 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:36.373029 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.374239 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.375346 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.375961 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.376083 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.378880 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:36.378885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:36.378885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:36.378897 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:36.380824 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.381237 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.381240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.381240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.381363 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.381596 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.381597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.381597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.381914 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:36.382064 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:36.382192 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 15:11:36.382323 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260923 15:11:36.388429 25434 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:36.389024 25434 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:36.398139 25434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3beef4bcb9c4d7a890744f220bbff00: Generated new TSK 0
I20260923 15:11:36.398346 25434 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:36.429270 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40079
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:36.518044 25446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:36.518280 25446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:36.518306 25446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:36.519461 25446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:36.519513 25446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:36.519543 25446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:36.520969 25446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:36.521183 25446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:36.521404 25446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:36.526681 25454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:36.526713 25452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:36.526703 25451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:36.526927 25446 server_base.cc:1034] running on GCE node
I20260923 15:11:36.527139 25446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:36.527406 25446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:36.528575 25446 hybrid_clock.cc:648] HybridClock initialized: now 1790176296528564 us; error 33 us; skew 500 ppm
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:36.531147 25446 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:36.531592 25446 webserver.cc:460] Webserver started at http://127.21.204.65:41771/ using document root <none> and password file <none>
I20260923 15:11:36.531847 25446 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:36.531893 25446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:36.532020 25446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:36.533057 25446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "7c63cc9676a745ee9711805d27a920a1"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "1eeb32535b2abe124ea84b31ce4d1215"
server_key_iv: "ef9b9c189c8675435497c4fb644fb3ff"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.533452 25446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "7c63cc9676a745ee9711805d27a920a1"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "1eeb32535b2abe124ea84b31ce4d1215"
server_key_iv: "ef9b9c189c8675435497c4fb644fb3ff"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.534925 25446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:36.535804 25461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.536139 25446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:36.536229 25446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "7c63cc9676a745ee9711805d27a920a1"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "1eeb32535b2abe124ea84b31ce4d1215"
server_key_iv: "ef9b9c189c8675435497c4fb644fb3ff"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.536343 25446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:36.549649 25446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:36.552047 25446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:36.552210 25446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:36.552587 25446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:36.552969 25446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:36.553007 25446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.553030 25446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:36.553043 25446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.559651 25446 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:37537
I20260923 15:11:36.559693 25574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:37537 every 8 connection(s)
I20260923 15:11:36.560039 25446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:36.564560 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25446
I20260923 15:11:36.564666 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:36.564862 22321 external_mini_cluster.cc:1428] Setting key 34c11879710094386482611be467383f
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:36.570312 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.560864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:55745 (local address 127.21.204.126:40079)
0923 15:11:36.560932 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:36.560934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:36.561166 (+ 232us) server_negotiation.cc:407] Connection header received
0923 15:11:36.561660 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:36.561661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:36.561675 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:36.561739 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:36.562774 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.563288 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.564424 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.564803 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.564886 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.567315 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:36.567317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:36.567317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:36.567327 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:36.568786 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.569241 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.569242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.569243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.569325 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.569589 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.569590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.569591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.569713 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:36.569793 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:36.570131 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 15:11:36.570218 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:11:36.570506 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.560832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40079 (local address 127.21.204.65:55745)
0923 15:11:36.561018 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:36.561062 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:36.561566 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:36.561860 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:36.561864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:36.562458 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:36.562659 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:36.562663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.563396 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.563397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:36.564306 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:36.564309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.564896 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.564918 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:36.565090 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.565757 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:36.565766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:36.567216 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:36.568891 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:36.568899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:36.568902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:36.569128 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:36.569396 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:36.569397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:36.569397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:36.569517 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:36.569785 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:36.569793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:36.569997 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 15:11:36.570108 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 15:11:36.570278 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
I20260923 15:11:36.571053 25575 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40079
I20260923 15:11:36.571174 25575 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:36.571377 25575 heartbeater.cc:511] Master 127.21.204.126:40079 requested a full tablet report, sending...
I20260923 15:11:36.571920 25384 ts_manager.cc:194] Registered new tserver with Master: 7c63cc9676a745ee9711805d27a920a1 (127.21.204.65:37537)
I20260923 15:11:36.572518 25384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:55745
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:36.592635 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40079
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:36.673544 25580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:36.673748 25580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:36.673772 25580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:36.675423 25580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:36.675480 25580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:36.675510 25580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:36.677336 25580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:36.677610 25580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:36.677858 25580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:36.683002 25588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:36.683127 25580 server_base.cc:1034] running on GCE node
W20260923 15:11:36.682969 25585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:36.683209 25586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:36.683424 25580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:36.683718 25580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:36.684862 25580 hybrid_clock.cc:648] HybridClock initialized: now 1790176296684841 us; error 37 us; skew 500 ppm
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:36.687345 25580 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:36.687826 25580 webserver.cc:460] Webserver started at http://127.21.204.66:40967/ using document root <none> and password file <none>
I20260923 15:11:36.688072 25580 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:36.688150 25580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:36.688300 25580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:36.689342 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "385a1b2bb3e547628e8c77b52706c365"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "326ff256bb30fbe8fa2ae6a3fdb4700f"
server_key_iv: "824af6f6cd2d1150ab0acfe88a31fc6f"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.689702 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "385a1b2bb3e547628e8c77b52706c365"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "326ff256bb30fbe8fa2ae6a3fdb4700f"
server_key_iv: "824af6f6cd2d1150ab0acfe88a31fc6f"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.691174 25580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:36.692091 25595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.692360 25580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:36.692471 25580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "385a1b2bb3e547628e8c77b52706c365"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "326ff256bb30fbe8fa2ae6a3fdb4700f"
server_key_iv: "824af6f6cd2d1150ab0acfe88a31fc6f"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.692560 25580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:36.703148 25580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:36.705008 25580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:36.705176 25580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:36.705423 25580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:36.705739 25580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:36.705801 25580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.705842 25580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:36.705870 25580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.712244 25580 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:37967
I20260923 15:11:36.712348 25708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:37967 every 8 connection(s)
I20260923 15:11:36.712641 25580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:36.717353 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25580
I20260923 15:11:36.717454 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:36.717674 22321 external_mini_cluster.cc:1428] Setting key 1845d87c911ad1c2d000cc89d79e5a25
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:36.722752 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.713445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:60293 (local address 127.21.204.126:40079)
0923 15:11:36.713534 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:36.713536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:36.713680 (+ 144us) server_negotiation.cc:407] Connection header received
0923 15:11:36.714235 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:36.714237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:36.714248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:36.714303 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:36.715255 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.715759 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.716895 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.717077 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.717127 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.719702 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:36.719704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:36.719704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:36.719714 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:36.721107 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.721640 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.721641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.721641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.721690 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.722027 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.722028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.722028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.722180 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:36.722249 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:36.722618 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 15:11:36.722686 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 15:11:36.723061 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.713375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40079 (local address 127.21.204.66:60293)
0923 15:11:36.713529 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:36.713577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:36.714135 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:36.714359 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:36.714363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:36.714963 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:36.715174 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:36.715178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.715885 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.715886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:36.716815 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:36.716818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.717159 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.717160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:36.717275 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.717944 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:36.717959 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:36.719587 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:36.721199 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:36.721207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:36.721209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:36.721558 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:36.721764 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:36.721765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:36.721766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:36.721952 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:36.722266 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:36.722273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:36.722464 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 15:11:36.722587 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 15:11:36.722773 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260923 15:11:36.723635 25709 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40079
I20260923 15:11:36.723753 25709 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:36.723978 25709 heartbeater.cc:511] Master 127.21.204.126:40079 requested a full tablet report, sending...
I20260923 15:11:36.724414 25384 ts_manager.cc:194] Registered new tserver with Master: 385a1b2bb3e547628e8c77b52706c365 (127.21.204.66:37967)
I20260923 15:11:36.724793 25384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:60293
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:36.748471 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40079
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:36.830765 25714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:36.830971 25714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:36.830994 25714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:36.832141 25714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:36.832191 25714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:36.832218 25714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:36.833730 25714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:36.833997 25714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:36.834255 25714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:11:36.839401 25714 server_base.cc:1034] running on GCE node
W20260923 15:11:36.839432 25720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:36.839432 25719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:36.839447 25722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:36.839848 25714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:36.840137 25714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:36.841316 25714 hybrid_clock.cc:648] HybridClock initialized: now 1790176296841297 us; error 34 us; skew 500 ppm
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:36.843873 25714 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:36.844310 25714 webserver.cc:460] Webserver started at http://127.21.204.67:33009/ using document root <none> and password file <none>
I20260923 15:11:36.844558 25714 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:36.844630 25714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:36.844779 25714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:36.845849 25714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "dc0c0d359cb64853a200c07eb51089d6"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "8dc034d35f248ed57f258a7b9f5320a6"
server_key_iv: "8cce876d7d0f8ceef37e43907c44e3db"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.846275 25714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "dc0c0d359cb64853a200c07eb51089d6"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "8dc034d35f248ed57f258a7b9f5320a6"
server_key_iv: "8cce876d7d0f8ceef37e43907c44e3db"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.847833 25714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:36.848626 25729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.848796 25714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:36.848898 25714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "dc0c0d359cb64853a200c07eb51089d6"
format_stamp: "Formatted at 2026-09-23 15:11:36 on dist-test-slave-t5lt"
server_key: "8dc034d35f248ed57f258a7b9f5320a6"
server_key_iv: "8cce876d7d0f8ceef37e43907c44e3db"
server_key_version: "encryptionkey@0"
I20260923 15:11:36.848980 25714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:36.873309 25714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:36.876022 25714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:36.876183 25714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:36.876421 25714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:36.876750 25714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:36.876806 25714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.876848 25714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:36.876875 25714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:36.882843 25714 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:45711
I20260923 15:11:36.882953 25842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:45711 every 8 connection(s)
I20260923 15:11:36.883232 25714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:36.883641 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25714
I20260923 15:11:36.883723 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:36.883913 22321 external_mini_cluster.cc:1428] Setting key a7ea1ef9750ea4ff550fa051b5790a8c
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:36.893505 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.884154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:34815 (local address 127.21.204.126:40079)
0923 15:11:36.884201 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:36.884203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:36.884420 (+ 217us) server_negotiation.cc:407] Connection header received
0923 15:11:36.885130 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:36.885131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:36.885145 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:36.885195 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:36.886298 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.886846 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.887958 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.888178 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.888274 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.890752 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:36.890754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:36.890754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:36.890763 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:36.892053 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.892515 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.892516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.892517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.892578 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.892840 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.892842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.892842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.892967 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:36.893047 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:36.893331 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 15:11:36.893413 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:11:36.893720 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.884108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40079 (local address 127.21.204.67:34815)
0923 15:11:36.884298 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:36.884350 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:36.885014 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:36.885265 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:36.885269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:36.885952 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:36.886178 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:36.886183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.886952 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.886953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:36.887838 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:36.887841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.888283 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.888285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:36.888418 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.889098 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:36.889107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:36.890581 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:36.892191 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:36.892198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:36.892209 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:36.892431 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:36.892646 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:36.892647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:36.892648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:36.892767 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:36.893038 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:36.893044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:36.893187 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 15:11:36.893307 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 15:11:36.893468 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260923 15:11:36.894244 25843 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40079
I20260923 15:11:36.894326 25843 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:36.894538 25843 heartbeater.cc:511] Master 127.21.204.126:40079 requested a full tablet report, sending...
I20260923 15:11:36.894963 25384 ts_manager.cc:194] Registered new tserver with Master: dc0c0d359cb64853a200c07eb51089d6 (127.21.204.67:45711)
I20260923 15:11:36.895445 25384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:34815
I20260923 15:11:36.895718 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176296, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:36.918428 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:36.911031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.21.204.126:40079)
0923 15:11:36.911122 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:36.911124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:36.911242 (+ 118us) server_negotiation.cc:407] Connection header received
0923 15:11:36.911346 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:36.911347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:36.911359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:36.911437 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:36.912042 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.912536 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.913301 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:36.913489 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:36.913594 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:36.915941 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:36.915942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:36.915942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:36.915948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:36.917148 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.917517 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.917518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.917519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.917580 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:36.917791 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:36.917792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:36.917792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:36.918027 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:36.918110 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:36.918293 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 15:11:36.918359 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 15:11:36.918671 25384 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48828
I20260923 15:11:36.919200 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25446
I20260923 15:11:36.924393 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25580
I20260923 15:11:36.928519 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25714
I20260923 15:11:36.933271 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25352
2026-09-23T15:11:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1913 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25855](info): set up 2 sockets
Sep 23 15:11:36 dist-test-slave-t5lt krb5kdc[25855](info): commencing operation
krb5kdc: starting...
W20260923 15:11:37.988792 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260923 15:11:38.254734 22321 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.179s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260923 15:11:38.415041 22321 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.160s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/test-user.keytab.
W20260923 15:11:39.118511 22321 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.703s user 0.000s sys 0.001s
Sep 23 15:11:39 dist-test-slave-t5lt krb5kdc[25855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
W20260923 15:11:40.406636 22321 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.204.126: real 1.265s user 0.000s sys 0.003s
I20260923 15:11:40.407073 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33041
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:33041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:40.492722 25868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:40.492926 25868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:40.492949 25868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:40.494161 25868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:40.494205 25868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:40.494220 25868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:40.494232 25868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:40.494244 25868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:40.495877 25868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:33041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:40.496151 25868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:40.496423 25868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:40.501403 25874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:40.501482 25876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:40.501410 25873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:40.502521 25868 server_base.cc:1034] running on GCE node
I20260923 15:11:40.502815 25868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:40.503118 25868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:40.504316 25868 hybrid_clock.cc:648] HybridClock initialized: now 1790176300504290 us; error 45 us; skew 500 ppm
Sep 23 15:11:40 dist-test-slave-t5lt krb5kdc[25855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176300, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:40.507104 25868 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:40.507634 25868 webserver.cc:460] Webserver started at http://127.21.204.126:38323/ using document root <none> and password file <none>
I20260923 15:11:40.507903 25868 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:40.507977 25868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:40.508131 25868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:40.509284 25868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "b2950ba9d13b4a4993b61b7023c2cfe5"
format_stamp: "Formatted at 2026-09-23 15:11:40 on dist-test-slave-t5lt"
server_key: "c391bd247e89fc676c1a4bbbd87f48cc"
server_key_iv: "c198d048c9b0593f5257eec8ec8fa490"
server_key_version: "encryptionkey@0"
I20260923 15:11:40.509663 25868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "b2950ba9d13b4a4993b61b7023c2cfe5"
format_stamp: "Formatted at 2026-09-23 15:11:40 on dist-test-slave-t5lt"
server_key: "c391bd247e89fc676c1a4bbbd87f48cc"
server_key_iv: "c198d048c9b0593f5257eec8ec8fa490"
server_key_version: "encryptionkey@0"
I20260923 15:11:40.511160 25868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:40.512104 25883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:40.512360 25868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:40.512480 25868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "b2950ba9d13b4a4993b61b7023c2cfe5"
format_stamp: "Formatted at 2026-09-23 15:11:40 on dist-test-slave-t5lt"
server_key: "c391bd247e89fc676c1a4bbbd87f48cc"
server_key_iv: "c198d048c9b0593f5257eec8ec8fa490"
server_key_version: "encryptionkey@0"
I20260923 15:11:40.512562 25868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:40.526755 25868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:40.528887 25868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:40.529065 25868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:40.534015 25868 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:33041
I20260923 15:11:40.534054 25935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:33041 every 8 connection(s)
I20260923 15:11:40.534601 25868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:40.535401 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:40.538329 25936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5: Bootstrap starting.
I20260923 15:11:40.539115 25936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:40.539469 25936 log.cc:824] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:40.540341 25936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5: No bootstrap required, opened a new log
I20260923 15:11:40.542239 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25868
I20260923 15:11:40.542114 25936 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33041 } }
I20260923 15:11:40.542302 25936 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:40.542348 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:40.542343 25936 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2950ba9d13b4a4993b61b7023c2cfe5, State: Initialized, Role: FOLLOWER
I20260923 15:11:40.542536 22321 external_mini_cluster.cc:1428] Setting key e9bb970e54a3d64d46306191f25562e6
I20260923 15:11:40.542511 25936 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33041 } }
I20260923 15:11:40.542604 25936 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:40.542646 25936 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:40.542698 25936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 15:11:40 dist-test-slave-t5lt krb5kdc[25855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:40.552090 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:40.543059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.21.204.126:33041)
0923 15:11:40.543279 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:40.543289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:40.543326 (+ 37us) server_negotiation.cc:407] Connection header received
0923 15:11:40.543421 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:40.543426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:40.543477 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:40.543610 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:40.544263 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:40.545142 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:40.545909 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:40.546335 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:40.546466 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:40.548704 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:40.548708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:40.548709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:40.548727 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:40.550661 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:40.550995 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:40.550997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:40.550997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:40.551080 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:40.551250 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:40.551251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:40.551251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:40.551481 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:40.551580 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:40.551825 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 15:11:40.551931 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260923 15:11:40.616705 25936 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33041 } }
I20260923 15:11:40.616878 25936 leader_election.cc:302] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2950ba9d13b4a4993b61b7023c2cfe5; no voters:
I20260923 15:11:40.617144 25936 leader_election.cc:288] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:40.617229 25941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:40.617496 25936 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:40.617520 25941 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [term 1 LEADER]: Becoming Leader. State: Replica: b2950ba9d13b4a4993b61b7023c2cfe5, State: Running, Role: LEADER
I20260923 15:11:40.617805 25941 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33041 } }
I20260923 15:11:40.618283 25943 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2950ba9d13b4a4993b61b7023c2cfe5. Latest consensus state: current_term: 1 leader_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33041 } } }
I20260923 15:11:40.618382 25943 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:40.618602 25950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:40.618656 25942 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2950ba9d13b4a4993b61b7023c2cfe5" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33041 } } }
I20260923 15:11:40.618994 25942 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:40.619272 25950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:40.621213 25950 catalog_manager.cc:1347] Generated new cluster ID: c726a53df39943f896354c814a7756f6
I20260923 15:11:40.621273 25950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:40.626729 25950 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:40.627342 25950 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:40.636958 25950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2950ba9d13b4a4993b61b7023c2cfe5: Generated new TSK 0
I20260923 15:11:40.637167 25950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
W20260923 15:11:41.773234 22321 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.204.65: real 1.133s user 0.000s sys 0.003s
I20260923 15:11:41.773625 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33041
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:41.862797 25962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:41.863006 25962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:41.863032 25962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:41.864276 25962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:41.864326 25962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:41.864352 25962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:41.866048 25962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:41.866330 25962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:41.866606 25962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:41.874100 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:41.874089 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:41.874089 25962 server_base.cc:1034] running on GCE node
W20260923 15:11:41.874089 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:41.874595 25962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:41.874912 25962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:41.876153 25962 hybrid_clock.cc:648] HybridClock initialized: now 1790176301876136 us; error 28 us; skew 500 ppm
Sep 23 15:11:41 dist-test-slave-t5lt krb5kdc[25855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176301, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:41.878947 25962 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:41.879465 25962 webserver.cc:460] Webserver started at http://127.21.204.65:36223/ using document root <none> and password file <none>
I20260923 15:11:41.879688 25962 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:41.879737 25962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:41.879838 25962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:41.880996 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "ea677d32e1974c7b9f41214a685981d1"
format_stamp: "Formatted at 2026-09-23 15:11:41 on dist-test-slave-t5lt"
server_key: "81af675534f2662a1c1d336ead14c29d"
server_key_iv: "73f9adab7e5772eb1637bb600b39215c"
server_key_version: "encryptionkey@0"
I20260923 15:11:41.881350 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "ea677d32e1974c7b9f41214a685981d1"
format_stamp: "Formatted at 2026-09-23 15:11:41 on dist-test-slave-t5lt"
server_key: "81af675534f2662a1c1d336ead14c29d"
server_key_iv: "73f9adab7e5772eb1637bb600b39215c"
server_key_version: "encryptionkey@0"
I20260923 15:11:41.882856 25962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:41.884032 25977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:41.884305 25962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:41.884395 25962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "ea677d32e1974c7b9f41214a685981d1"
format_stamp: "Formatted at 2026-09-23 15:11:41 on dist-test-slave-t5lt"
server_key: "81af675534f2662a1c1d336ead14c29d"
server_key_iv: "73f9adab7e5772eb1637bb600b39215c"
server_key_version: "encryptionkey@0"
I20260923 15:11:41.884456 25962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:41.907011 25962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:41.909467 25962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:41.909634 25962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:41.909967 25962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:41.910416 25962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:41.910465 25962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:41.910501 25962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:41.910601 25962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:41.919903 25962 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:36039
I20260923 15:11:41.920197 26090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:36039 every 8 connection(s)
I20260923 15:11:41.920401 25962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
Sep 23 15:11:41 dist-test-slave-t5lt krb5kdc[25855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176301, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:41.929800 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 25962
I20260923 15:11:41.929976 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:41.930219 22321 external_mini_cluster.cc:1428] Setting key ab854d7f1ed84c0036371944873ee8b7
I20260923 15:11:41.932636 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:41.921253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:46383 (local address 127.21.204.126:33041)
0923 15:11:41.921953 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:41.921955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:41.921968 (+ 13us) server_negotiation.cc:407] Connection header received
0923 15:11:41.922782 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:41.922783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:41.922797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:41.922862 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:41.923847 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:41.924373 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:41.925503 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:41.925713 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:41.925776 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:41.929006 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:41.929007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:41.929008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:41.929015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:41.930652 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:41.931203 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:41.931204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:41.931204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:41.931307 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:41.931891 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:41.931892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:41.931892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:41.932015 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:41.932080 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:41.932475 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 15:11:41.932557 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":81,"threads_started":1}
I20260923 15:11:41.932919 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:41.921191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33041 (local address 127.21.204.65:46383)
0923 15:11:41.921405 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:41.921451 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:41.922656 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:41.922934 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:41.922939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:41.923346 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:41.923663 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:41.923668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:41.924549 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:41.924550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:41.925405 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:41.925408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:41.925806 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:41.925807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:41.926002 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:41.927280 (+ 1278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:41.927289 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:41.928849 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:41.930736 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:41.930745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:41.930747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:41.931082 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:41.931392 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:41.931393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:41.931393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:41.931726 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:41.932095 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:41.932101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:41.932289 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 15:11:41.932410 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 15:11:41.932613 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260923 15:11:41.933974 26091 heartbeater.cc:347] Connected to a master server at 127.21.204.126:33041
I20260923 15:11:41.934101 26091 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:41.934296 26091 heartbeater.cc:511] Master 127.21.204.126:33041 requested a full tablet report, sending...
I20260923 15:11:41.935061 25900 ts_manager.cc:194] Registered new tserver with Master: ea677d32e1974c7b9f41214a685981d1 (127.21.204.65:36039)
I20260923 15:11:41.935842 25900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:46383
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
W20260923 15:11:42.373483 22321 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.204.66: real 0.442s user 0.000s sys 0.003s
I20260923 15:11:42.373917 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33041
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:42.460778 26097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:42.461016 26097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:42.461054 26097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:42.462304 26097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:42.462368 26097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:42.462410 26097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:42.463924 26097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:42.464207 26097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:42.464437 26097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:42.469581 26105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:42.469834 26103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:42.469841 26102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:42.469973 26097 server_base.cc:1034] running on GCE node
I20260923 15:11:42.470170 26097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:42.470371 26097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:42.471525 26097 hybrid_clock.cc:648] HybridClock initialized: now 1790176302471510 us; error 31 us; skew 500 ppm
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[25855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:42.474160 26097 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:42.474627 26097 webserver.cc:460] Webserver started at http://127.21.204.66:41447/ using document root <none> and password file <none>
I20260923 15:11:42.474871 26097 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:42.474941 26097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:42.475090 26097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:42.476114 26097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "e4eb5113e146417181e276ba32c653be"
format_stamp: "Formatted at 2026-09-23 15:11:42 on dist-test-slave-t5lt"
server_key: "05b0c1158fbb762512ddc9b805780721"
server_key_iv: "2976cf8d9fa3db4d0d082ebe98fdb281"
server_key_version: "encryptionkey@0"
I20260923 15:11:42.476577 26097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "e4eb5113e146417181e276ba32c653be"
format_stamp: "Formatted at 2026-09-23 15:11:42 on dist-test-slave-t5lt"
server_key: "05b0c1158fbb762512ddc9b805780721"
server_key_iv: "2976cf8d9fa3db4d0d082ebe98fdb281"
server_key_version: "encryptionkey@0"
I20260923 15:11:42.478401 26097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:42.479333 26112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:42.479630 26097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:42.479780 26097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "e4eb5113e146417181e276ba32c653be"
format_stamp: "Formatted at 2026-09-23 15:11:42 on dist-test-slave-t5lt"
server_key: "05b0c1158fbb762512ddc9b805780721"
server_key_iv: "2976cf8d9fa3db4d0d082ebe98fdb281"
server_key_version: "encryptionkey@0"
I20260923 15:11:42.479866 26097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:42.501092 26097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:42.503455 26097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:42.503578 26097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:42.503758 26097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:42.504078 26097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:42.504136 26097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:42.504182 26097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:42.504215 26097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:42.510288 26097 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:42963
I20260923 15:11:42.510361 26225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:42963 every 8 connection(s)
I20260923 15:11:42.510710 26097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[25855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:42.519225 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 26097
I20260923 15:11:42.519364 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:42.519662 22321 external_mini_cluster.cc:1428] Setting key 2f9aeb3fa5915c0f38f7e3922f522d0b
I20260923 15:11:42.522024 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:42.511602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:46789 (local address 127.21.204.126:33041)
0923 15:11:42.511743 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:42.511746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:42.512080 (+ 334us) server_negotiation.cc:407] Connection header received
0923 15:11:42.513087 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:42.513088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:42.513105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:42.513168 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:42.513815 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.514402 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.515809 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.516042 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.516161 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:42.518850 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:42.518852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:42.518852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:42.518861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:42.520559 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.520996 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.520997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.520998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.521062 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.521333 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.521333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.521334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.521452 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:42.521528 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:42.521820 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 15:11:42.521928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260923 15:11:42.522238 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:42.511685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33041 (local address 127.21.204.66:46789)
0923 15:11:42.511924 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:42.511971 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:42.512952 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:42.513253 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:42.513263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:42.513514 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:42.513705 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:42.513718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.514513 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.514515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:42.515696 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:42.515699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.516187 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.516188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:42.516311 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:42.516891 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:42.516900 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:42.518726 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:42.520652 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:42.520660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:42.520663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:42.520903 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:42.521132 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:42.521133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:42.521133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:42.521256 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:42.521520 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:42.521527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:42.521674 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 15:11:42.521786 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 15:11:42.521972 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260923 15:11:42.522688 26226 heartbeater.cc:347] Connected to a master server at 127.21.204.126:33041
I20260923 15:11:42.522785 26226 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:42.522934 26226 heartbeater.cc:511] Master 127.21.204.126:33041 requested a full tablet report, sending...
I20260923 15:11:42.523315 25900 ts_manager.cc:194] Registered new tserver with Master: e4eb5113e146417181e276ba32c653be (127.21.204.66:42963)
I20260923 15:11:42.523715 25900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:46789
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:42.550253 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33041
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:42.635092 26232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:42.635288 26232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:42.635311 26232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:42.636642 26232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:42.636698 26232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:42.636727 26232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:42.638221 26232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:33041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:42.638403 26232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:42.638582 26232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:42.643592 26238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:42.643574 26240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:42.643580 26237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:42.643664 26232 server_base.cc:1034] running on GCE node
I20260923 15:11:42.644035 26232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:42.644246 26232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:42.645396 26232 hybrid_clock.cc:648] HybridClock initialized: now 1790176302645386 us; error 29 us; skew 500 ppm
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[25855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:42.648094 26232 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:42.648592 26232 webserver.cc:460] Webserver started at http://127.21.204.67:43631/ using document root <none> and password file <none>
I20260923 15:11:42.648851 26232 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:42.648922 26232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:42.649075 26232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:42.650214 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "8cfcee3361ce43e0b26f9cd494178d19"
format_stamp: "Formatted at 2026-09-23 15:11:42 on dist-test-slave-t5lt"
server_key: "5f24812592d48dd80bb24de20c100510"
server_key_iv: "eb6e49659fca1a244995b73c81cb6c61"
server_key_version: "encryptionkey@0"
I20260923 15:11:42.650585 26232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "8cfcee3361ce43e0b26f9cd494178d19"
format_stamp: "Formatted at 2026-09-23 15:11:42 on dist-test-slave-t5lt"
server_key: "5f24812592d48dd80bb24de20c100510"
server_key_iv: "eb6e49659fca1a244995b73c81cb6c61"
server_key_version: "encryptionkey@0"
I20260923 15:11:42.651989 26232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:11:42.652876 26247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:42.653134 26232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:42.653261 26232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "8cfcee3361ce43e0b26f9cd494178d19"
format_stamp: "Formatted at 2026-09-23 15:11:42 on dist-test-slave-t5lt"
server_key: "5f24812592d48dd80bb24de20c100510"
server_key_iv: "eb6e49659fca1a244995b73c81cb6c61"
server_key_version: "encryptionkey@0"
I20260923 15:11:42.653340 26232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:42.688931 26232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:42.691627 26232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:42.691824 26232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:42.692091 26232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:42.692425 26232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:42.692483 26232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:42.692536 26232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:42.692561 26232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:42.698596 26232 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:37015
I20260923 15:11:42.698645 26360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:37015 every 8 connection(s)
I20260923 15:11:42.698961 26232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[25855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:42.705772 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 26232
I20260923 15:11:42.705971 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:42.706188 22321 external_mini_cluster.cc:1428] Setting key 750eab0fb8fea7f2219867c8263a2f3a
I20260923 15:11:42.709405 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:42.699757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:39869 (local address 127.21.204.126:33041)
0923 15:11:42.699815 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:42.699817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:42.699988 (+ 171us) server_negotiation.cc:407] Connection header received
0923 15:11:42.701010 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:42.701011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:42.701021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:42.701075 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:42.702061 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.702612 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.703724 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.703961 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.704027 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:42.706453 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:42.706454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:42.706455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:42.706464 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:42.707953 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.708399 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.708400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.708400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.708462 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.708732 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.708733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.708733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.708852 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:42.708931 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:42.709206 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 15:11:42.709315 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:42.709606 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:42.699710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33041 (local address 127.21.204.67:39869)
0923 15:11:42.699866 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:42.699905 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:42.700913 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:42.701151 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:42.701154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:42.701658 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:42.701862 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:42.701867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.702781 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.702782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:42.703585 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:42.703588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.704074 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.704075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:42.704209 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:42.704700 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:42.704709 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:42.706332 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:42.708083 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:42.708089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:42.708092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:42.708317 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:42.708538 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:42.708539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:42.708539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:42.708652 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:42.708921 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:42.708927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:42.709074 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 15:11:42.709184 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 15:11:42.709355 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260923 15:11:42.710134 26361 heartbeater.cc:347] Connected to a master server at 127.21.204.126:33041
I20260923 15:11:42.710235 26361 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:42.710381 26361 heartbeater.cc:511] Master 127.21.204.126:33041 requested a full tablet report, sending...
I20260923 15:11:42.710853 25900 ts_manager.cc:194] Registered new tserver with Master: 8cfcee3361ce43e0b26f9cd494178d19 (127.21.204.67:37015)
I20260923 15:11:42.711278 25900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:39869
I20260923 15:11:42.718425 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[25855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[25855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:42.744063 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:42.735099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.21.204.65:36039)
0923 15:11:42.735248 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:42.735255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:42.735274 (+ 19us) server_negotiation.cc:407] Connection header received
0923 15:11:42.735302 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:42.735304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:42.735318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:42.735394 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:42.736121 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.736849 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.737703 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.738092 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.738158 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:42.740781 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:42.740790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:42.740790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:42.740798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:42.742516 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.742903 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.742906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.742906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.742974 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.743238 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.743239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.743239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.743382 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:42.743516 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:42.743895 (+ 379us) server_negotiation.cc:299] Negotiation successful
0923 15:11:42.743982 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[25855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:42.754642 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:42.745772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.21.204.126:33041)
0923 15:11:42.745869 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:42.745871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:42.745942 (+ 71us) server_negotiation.cc:407] Connection header received
0923 15:11:42.745975 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:42.745976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:42.745986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:42.746040 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:42.746571 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.747083 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.747899 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:42.748108 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:42.748172 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:42.751688 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:42.751690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:42.751690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:42.751698 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:42.753148 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.753555 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.753556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.753557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.753615 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:42.753873 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:42.753959 (+ 86us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:42.753959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:42.754135 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:42.754223 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:42.754414 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 15:11:42.754480 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 15:11:42.754931 25900 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54810
I20260923 15:11:42.755609 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25962
I20260923 15:11:42.760679 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 26097
I20260923 15:11:42.765331 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 26232
I20260923 15:11:42.769582 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 25868
2026-09-23T15:11:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5835 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[26373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[26373](info): set up 2 sockets
Sep 23 15:11:42 dist-test-slave-t5lt krb5kdc[26373](info): commencing operation
krb5kdc: starting...
W20260923 15:11:43.867295 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:43 dist-test-slave-t5lt krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:44.003515 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40301
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:44.093928 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:44.094142 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:44.094166 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:44.095381 26386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:44.095422 26386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:44.095438 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:44.095450 26386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:44.095463 26386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:44.096972 26386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:40301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:44.097225 26386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:44.097464 26386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:44.102348 26394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:44.102375 26392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:44.102526 26386 server_base.cc:1034] running on GCE node
W20260923 15:11:44.102502 26391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:44.102936 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:44.103267 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:44.104447 26386 hybrid_clock.cc:648] HybridClock initialized: now 1790176304104432 us; error 46 us; skew 500 ppm
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176304, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:44.106930 26386 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:44.107267 26386 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:44.107343 26386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:44.107493 26386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:44.108705 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "21c84eb659ce478694067ce93a1fee72"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "db8aea29948e3af9cb547a87dcc070cf"
server_key_iv: "20924d1bb385e2aed16e689d55e231a4"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.109159 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "21c84eb659ce478694067ce93a1fee72"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "db8aea29948e3af9cb547a87dcc070cf"
server_key_iv: "20924d1bb385e2aed16e689d55e231a4"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.110584 26386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:44.111343 26400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.111636 26386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:44.111749 26386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "21c84eb659ce478694067ce93a1fee72"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "db8aea29948e3af9cb547a87dcc070cf"
server_key_iv: "20924d1bb385e2aed16e689d55e231a4"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.111828 26386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:44.143903 26386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:44.146255 26386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:44.146420 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:44.150434 26386 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:40301
I20260923 15:11:44.150462 26452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:40301 every 8 connection(s)
I20260923 15:11:44.150765 26386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:44.151912 26453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:44.154804 26453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72: Bootstrap starting.
I20260923 15:11:44.155516 26453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:44.155861 26453 log.cc:824] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:44.156672 26453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72: No bootstrap required, opened a new log
I20260923 15:11:44.158526 26453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c84eb659ce478694067ce93a1fee72" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40301 } }
I20260923 15:11:44.158689 26453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:44.158716 26453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c84eb659ce478694067ce93a1fee72, State: Initialized, Role: FOLLOWER
I20260923 15:11:44.158887 26453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c84eb659ce478694067ce93a1fee72" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40301 } }
I20260923 15:11:44.158978 26453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:44.159005 26453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:44.159091 26453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:44.159919 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 26386
I20260923 15:11:44.160055 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:44.160084 26453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c84eb659ce478694067ce93a1fee72" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40301 } }
I20260923 15:11:44.160209 26453 leader_election.cc:302] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21c84eb659ce478694067ce93a1fee72; no voters:
I20260923 15:11:44.160290 22321 external_mini_cluster.cc:1428] Setting key f1a0c003bea410d3e17e50adf6ea5ae5
I20260923 15:11:44.160408 26453 leader_election.cc:288] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:44.160446 26456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:44.160597 26456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [term 1 LEADER]: Becoming Leader. State: Replica: 21c84eb659ce478694067ce93a1fee72, State: Running, Role: LEADER
I20260923 15:11:44.160702 26456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c84eb659ce478694067ce93a1fee72" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40301 } }
I20260923 15:11:44.160730 26453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:44.162111 26458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21c84eb659ce478694067ce93a1fee72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c84eb659ce478694067ce93a1fee72" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40301 } } }
I20260923 15:11:44.162212 26458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:44.162648 26473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:44.162756 26457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21c84eb659ce478694067ce93a1fee72. Latest consensus state: current_term: 1 leader_uuid: "21c84eb659ce478694067ce93a1fee72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c84eb659ce478694067ce93a1fee72" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40301 } } }
I20260923 15:11:44.162989 26457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:44.163209 26473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:44.165079 26473 catalog_manager.cc:1347] Generated new cluster ID: 9b5780804f894445ae139f604181e66f
I20260923 15:11:44.165156 26473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:44.174492 26473 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:44.174705 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.161124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.21.204.126:40301)
0923 15:11:44.161292 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.161300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.161359 (+ 59us) server_negotiation.cc:407] Connection header received
0923 15:11:44.161478 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.161484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.161536 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.161754 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:44.163167 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.164827 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.166062 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.166670 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.166790 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.169957 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:44.169961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:44.169961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:44.169975 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:44.172810 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.173249 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.173253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.173253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.173373 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.173598 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.173599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.173599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.173963 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:44.174131 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.174258 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.174419 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260923 15:11:44.175166 26473 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:44.179888 26473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72: Generated new TSK 0
I20260923 15:11:44.180090 26473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:44.211220 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40301
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:44.293604 26479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:44.293803 26479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:44.293828 26479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:44.294983 26479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:44.295024 26479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:44.295053 26479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:44.296641 26479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:44.296882 26479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:44.297117 26479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:44.302539 26487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:44.302548 26485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:44.302544 26484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:44.302913 26479 server_base.cc:1034] running on GCE node
I20260923 15:11:44.303140 26479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:44.303385 26479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:44.304596 26479 hybrid_clock.cc:648] HybridClock initialized: now 1790176304304591 us; error 38 us; skew 500 ppm
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176304, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:44.307421 26479 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:44.307767 26479 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:44.307839 26479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:44.307989 26479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:44.309042 26479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "e75718f341804d5eaefeaf28da35cccb"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "9ad62c524061a3f6168f7662ea8d1a5b"
server_key_iv: "b055894c5be44e0e8e213cf7f9307df3"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.309446 26479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "e75718f341804d5eaefeaf28da35cccb"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "9ad62c524061a3f6168f7662ea8d1a5b"
server_key_iv: "b055894c5be44e0e8e213cf7f9307df3"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.310868 26479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:44.311748 26493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.311997 26479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:44.312134 26479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "e75718f341804d5eaefeaf28da35cccb"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "9ad62c524061a3f6168f7662ea8d1a5b"
server_key_iv: "b055894c5be44e0e8e213cf7f9307df3"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.312227 26479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:44.327929 26479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:44.330374 26479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:44.330555 26479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:44.330806 26479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:44.331149 26479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:44.331215 26479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.331261 26479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:44.331287 26479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.337185 26479 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:37489
I20260923 15:11:44.337271 26606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:37489 every 8 connection(s)
I20260923 15:11:44.337545 26479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176304, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:44.346984 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 26479
I20260923 15:11:44.347126 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:44.347359 22321 external_mini_cluster.cc:1428] Setting key b0fc06786a4b89dc3ca55c48c0a73071
I20260923 15:11:44.350477 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.338366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:36481 (local address 127.21.204.126:40301)
0923 15:11:44.338414 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.338416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.338663 (+ 247us) server_negotiation.cc:407] Connection header received
0923 15:11:44.339207 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.339208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.339225 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.339280 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:44.342317 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.342826 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.344157 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.344384 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.344479 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.346900 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:44.346902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:44.346904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:44.346922 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:44.348683 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.349136 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.349138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.349138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.349202 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.349489 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.349490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.349490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.349611 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:44.349698 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.350183 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.350377 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:11:44.350749 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.338334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40301 (local address 127.21.204.65:36481)
0923 15:11:44.338536 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.338573 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.339091 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.339371 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.339375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.339636 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:44.342159 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.342167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.342938 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.342939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.344036 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.344038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.344493 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.344494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.344622 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.345188 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:44.345204 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:44.346776 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:44.348779 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:44.348789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:44.348791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:44.349028 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:44.349296 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:44.349297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:44.349297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:44.349415 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:44.349689 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:44.349698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:44.350026 (+ 328us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.350158 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.350439 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260923 15:11:44.351223 26607 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40301
I20260923 15:11:44.351347 26607 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:44.351504 26607 heartbeater.cc:511] Master 127.21.204.126:40301 requested a full tablet report, sending...
I20260923 15:11:44.352051 26417 ts_manager.cc:194] Registered new tserver with Master: e75718f341804d5eaefeaf28da35cccb (127.21.204.65:37489)
I20260923 15:11:44.352675 26417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:36481
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:44.378901 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40301
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:44.464247 26612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:44.464443 26612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:44.464466 26612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:44.465641 26612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:44.465682 26612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:44.465709 26612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:44.467296 26612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:44.467554 26612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:44.467816 26612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:44.472889 26618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:44.472880 26620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:44.473253 26617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:44.473372 26612 server_base.cc:1034] running on GCE node
I20260923 15:11:44.473595 26612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:44.473940 26612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:44.475122 26612 hybrid_clock.cc:648] HybridClock initialized: now 1790176304475088 us; error 50 us; skew 500 ppm
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176304, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:44.478013 26612 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:44.478454 26612 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:44.478500 26612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:44.478596 26612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:44.479586 26612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "6c8a3fc7e86345adae498d355e6ad4af"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "bef152c840898e4f9b6d407a3bb029c0"
server_key_iv: "4597f96ecaa6f06f6991f434c91dfffe"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.479887 26612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "6c8a3fc7e86345adae498d355e6ad4af"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "bef152c840898e4f9b6d407a3bb029c0"
server_key_iv: "4597f96ecaa6f06f6991f434c91dfffe"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.481240 26612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:11:44.482112 26626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.482324 26612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:44.482425 26612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "6c8a3fc7e86345adae498d355e6ad4af"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "bef152c840898e4f9b6d407a3bb029c0"
server_key_iv: "4597f96ecaa6f06f6991f434c91dfffe"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.482484 26612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:44.496167 26612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:44.498260 26612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:44.498373 26612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:44.498562 26612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:44.498983 26612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:44.499035 26612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.499065 26612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:44.499078 26612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.504828 26612 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:36595
I20260923 15:11:44.504916 26739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:36595 every 8 connection(s)
I20260923 15:11:44.505168 26612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176304, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:44.514407 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.506049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:34549 (local address 127.21.204.126:40301)
0923 15:11:44.506129 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.506131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.506265 (+ 134us) server_negotiation.cc:407] Connection header received
0923 15:11:44.506732 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.506733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.506742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.506790 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:44.507710 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.508214 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.509235 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.509429 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.509486 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.511746 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:44.511747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:44.511747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:44.511754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:44.512990 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.513371 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.513372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.513373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.513422 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.513636 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.513636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.513637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.513738 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:44.513788 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.514231 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.514343 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 15:11:44.514547 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.505932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40301 (local address 127.21.204.66:34549)
0923 15:11:44.506132 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.506181 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.506641 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.506848 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.506851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.507116 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:44.507605 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.507610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.508317 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.508318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.509148 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.509150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.509527 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.509528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.509645 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.510242 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:44.510250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:44.511629 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:44.513072 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:44.513078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:44.513080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:44.513301 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:44.513484 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:44.513484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:44.513485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:44.513577 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:44.513836 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:44.513843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:44.514062 (+ 219us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.514202 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.514339 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260923 15:11:44.514704 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 26612
I20260923 15:11:44.514814 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:44.515053 22321 external_mini_cluster.cc:1428] Setting key 94db78e26aa3a465b1476a50119a03ea
I20260923 15:11:44.515097 26740 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40301
I20260923 15:11:44.515198 26740 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:44.515370 26740 heartbeater.cc:511] Master 127.21.204.126:40301 requested a full tablet report, sending...
I20260923 15:11:44.515960 26417 ts_manager.cc:194] Registered new tserver with Master: 6c8a3fc7e86345adae498d355e6ad4af (127.21.204.66:36595)
I20260923 15:11:44.516495 26417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:34549
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:44.544494 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40301
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:44.630209 26745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:44.630460 26745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:44.630503 26745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:44.631831 26745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:44.631908 26745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:44.631960 26745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:44.633654 26745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:44.633987 26745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:44.634249 26745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:44.639811 26751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:44.639811 26753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:44.639883 26750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:44.639842 26745 server_base.cc:1034] running on GCE node
I20260923 15:11:44.640245 26745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:44.640488 26745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:44.641644 26745 hybrid_clock.cc:648] HybridClock initialized: now 1790176304641630 us; error 29 us; skew 500 ppm
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176304, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:44.644150 26745 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:44.644444 26745 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:44.644491 26745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:44.644575 26745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:44.645543 26745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "b6cb8506bbf047e2aaea582892fd24af"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "10df92ad3dcea561c446636727ba351c"
server_key_iv: "b6860b5e5b9061d8ca4e2b85a0b19ddb"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.645843 26745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "b6cb8506bbf047e2aaea582892fd24af"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "10df92ad3dcea561c446636727ba351c"
server_key_iv: "b6860b5e5b9061d8ca4e2b85a0b19ddb"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.647137 26745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:44.647996 26759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.648221 26745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:44.648324 26745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "b6cb8506bbf047e2aaea582892fd24af"
format_stamp: "Formatted at 2026-09-23 15:11:44 on dist-test-slave-t5lt"
server_key: "10df92ad3dcea561c446636727ba351c"
server_key_iv: "b6860b5e5b9061d8ca4e2b85a0b19ddb"
server_key_version: "encryptionkey@0"
I20260923 15:11:44.648411 26745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:44.671738 26745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:44.674018 26745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:44.674193 26745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:44.674450 26745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:44.674847 26745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:44.674904 26745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.674948 26745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:44.674974 26745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:44.681133 26745 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:41355
I20260923 15:11:44.681329 26872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:41355 every 8 connection(s)
I20260923 15:11:44.681506 26745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:11:44 dist-test-slave-t5lt krb5kdc[26373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176304, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:44.689942 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 26745
I20260923 15:11:44.690052 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:44.690265 22321 external_mini_cluster.cc:1428] Setting key 3af5b88717e48f4bee6c494d0d901f36
I20260923 15:11:44.691831 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.682354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:32983 (local address 127.21.204.126:40301)
0923 15:11:44.682401 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.682403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.682594 (+ 191us) server_negotiation.cc:407] Connection header received
0923 15:11:44.683067 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.683068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.683079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.683145 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:44.684056 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.684554 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.685660 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.685857 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.685971 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.688332 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:44.688333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:44.688334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:44.688348 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:44.689990 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.690527 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.690529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.690529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.690596 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.690959 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.690960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.690960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.691100 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:44.691155 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.691493 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.691748 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:11:44.692083 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.682317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40301 (local address 127.21.204.67:32983)
0923 15:11:44.682468 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.682511 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.682977 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.683206 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.683210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.683463 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:44.683961 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.683966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.684645 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.684645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.685585 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.685588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.686007 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.686009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.686147 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.686741 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:44.686755 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:44.688202 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:44.690087 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:44.690094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:44.690096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:44.690437 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:44.690700 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:44.690701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:44.690701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:44.690882 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:44.691175 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:44.691181 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:44.691348 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.691452 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.691580 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260923 15:11:44.692773 26873 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40301
I20260923 15:11:44.692863 26873 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:44.692981 26873 heartbeater.cc:511] Master 127.21.204.126:40301 requested a full tablet report, sending...
I20260923 15:11:44.693367 26417 ts_manager.cc:194] Registered new tserver with Master: b6cb8506bbf047e2aaea582892fd24af (127.21.204.67:41355)
I20260923 15:11:44.693740 26417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:32983
I20260923 15:11:44.702564 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:11:44.711627 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.703982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.21.204.126:40301)
0923 15:11:44.704083 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.704085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.704094 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:44.704216 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.704217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.704228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.704293 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:44.705135 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.705651 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.706566 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.706798 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.706864 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.708350 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:44.708351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:44.708351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:44.708358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:44.710066 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.710449 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.710450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.710451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.710513 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:44.710777 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:44.710778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:44.710778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:44.710945 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:44.711033 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.711246 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.711317 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 15:11:44.713613 26417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:44.715528 26417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:44.722851 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.718662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.21.204.65:37489)
0923 15:11:44.719097 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.719101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.719111 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:44.719176 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.719178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.719192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.719270 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:44.719491 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.720187 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.721799 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.722373 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.722468 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.722527 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.722709 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.722754 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":81,"threads_started":1}
I20260923 15:11:44.722894 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.718246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:36595 (local address 127.0.0.1:59306)
0923 15:11:44.718471 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.718817 (+ 346us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.718881 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.719170 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.719171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.719173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:44.719236 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.719238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.720492 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.720493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.721674 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.721677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.722648 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.722649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.722709 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.722712 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.722758 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.722792 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":41,"thread_start_us":93,"threads_started":1}
I20260923 15:11:44.722880 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.718502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37489 (local address 127.0.0.1:49654)
0923 15:11:44.718981 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.718990 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.719024 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.719357 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.719358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.719360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:44.719397 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.719398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.720293 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.720294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.721708 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.721710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.722481 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.722482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.722565 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.722586 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.722686 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.722777 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":94,"thread_start_us":67,"threads_started":1}
I20260923 15:11:44.722893 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.718307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:41355 (local address 127.0.0.1:50450)
0923 15:11:44.718658 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.718710 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.718807 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.719107 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.719110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.719112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:44.719186 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.719188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.720508 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.720509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.721481 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.721484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.722550 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.722551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.722639 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.722643 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.722692 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.722781 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260923 15:11:44.723384 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.718429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.21.204.67:41355)
0923 15:11:44.718628 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.718632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.718787 (+ 155us) server_negotiation.cc:407] Connection header received
0923 15:11:44.718932 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.718935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.718954 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.719103 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:44.719247 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.720387 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.721574 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.722376 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.722450 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.722512 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.723255 (+ 743us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.723304 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260923 15:11:44.723400 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.718368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.21.204.66:36595)
0923 15:11:44.718540 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.718545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.718874 (+ 329us) server_negotiation.cc:407] Connection header received
0923 15:11:44.718969 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.718973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.718990 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.719066 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:44.719303 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.720386 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.721784 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.722505 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.722566 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.722616 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.723287 (+ 671us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.723326 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260923 15:11:44.724164 26807 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1 (DEFAULT_TABLE table=test-table [id=9f3cce61ffef426a8d9e8cc885946ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:44.724191 26541 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1 (DEFAULT_TABLE table=test-table [id=9f3cce61ffef426a8d9e8cc885946ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:44.724187 26674 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1 (DEFAULT_TABLE table=test-table [id=9f3cce61ffef426a8d9e8cc885946ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:44.724519 26807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:44.724516 26674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:44.724516 26541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:44.727361 26890 tablet_bootstrap.cc:492] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Bootstrap starting.
I20260923 15:11:44.727805 26891 tablet_bootstrap.cc:492] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb: Bootstrap starting.
I20260923 15:11:44.727957 26892 tablet_bootstrap.cc:492] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: Bootstrap starting.
I20260923 15:11:44.728170 26890 tablet_bootstrap.cc:654] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:44.728492 26891 tablet_bootstrap.cc:654] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:44.728534 26892 tablet_bootstrap.cc:654] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:44.728752 26890 log.cc:824] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:44.729135 26892 log.cc:824] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:44.729142 26891 log.cc:824] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:44.729759 26890 tablet_bootstrap.cc:492] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: No bootstrap required, opened a new log
I20260923 15:11:44.729841 26890 ts_tablet_manager.cc:1397] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:11:44.730149 26892 tablet_bootstrap.cc:492] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: No bootstrap required, opened a new log
I20260923 15:11:44.730165 26891 tablet_bootstrap.cc:492] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb: No bootstrap required, opened a new log
I20260923 15:11:44.730219 26892 ts_tablet_manager.cc:1397] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:44.730247 26891 ts_tablet_manager.cc:1397] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:11:44.731696 26891 raft_consensus.cc:348] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.731881 26891 raft_consensus.cc:374] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:44.731932 26891 raft_consensus.cc:729] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75718f341804d5eaefeaf28da35cccb, State: Initialized, Role: FOLLOWER
I20260923 15:11:44.731845 26890 raft_consensus.cc:348] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.732004 26890 raft_consensus.cc:374] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:44.732043 26890 raft_consensus.cc:729] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cb8506bbf047e2aaea582892fd24af, State: Initialized, Role: FOLLOWER
I20260923 15:11:44.732071 26891 consensus_queue.cc:260] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.732167 26890 consensus_queue.cc:260] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.732118 26892 raft_consensus.cc:348] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.732285 26892 raft_consensus.cc:374] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:44.732335 26892 raft_consensus.cc:729] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8a3fc7e86345adae498d355e6ad4af, State: Initialized, Role: FOLLOWER
I20260923 15:11:44.732403 26891 ts_tablet_manager.cc:1428] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:11:44.732466 26890 ts_tablet_manager.cc:1428] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:11:44.732447 26892 consensus_queue.cc:260] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.732527 26607 heartbeater.cc:503] Master 127.21.204.126:40301 was elected leader, sending a full tablet report...
I20260923 15:11:44.732583 26873 heartbeater.cc:503] Master 127.21.204.126:40301 was elected leader, sending a full tablet report...
I20260923 15:11:44.732693 26892 ts_tablet_manager.cc:1428] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 15:11:44.732812 26740 heartbeater.cc:503] Master 127.21.204.126:40301 was elected leader, sending a full tablet report...
W20260923 15:11:44.755728 26741 tablet.cc:2369] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:44.838337 26608 tablet.cc:2369] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:44.892959 26897 raft_consensus.cc:482] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:44.893106 26897 raft_consensus.cc:504] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.893479 26897 leader_election.cc:288] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c8a3fc7e86345adae498d355e6ad4af (127.21.204.66:36595), e75718f341804d5eaefeaf28da35cccb (127.21.204.65:37489)
I20260923 15:11:44.897042 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.893763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:50709 (local address 127.21.204.66:36595)
0923 15:11:44.893823 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.893826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.894025 (+ 199us) server_negotiation.cc:407] Connection header received
0923 15:11:44.894137 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.894138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.894149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.894195 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:44.894416 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.895004 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.896142 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.896669 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.896755 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.896766 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.896936 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.896978 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 15:11:44.897042 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.893684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:36595 (local address 127.21.204.67:50709)
0923 15:11:44.893966 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.893979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.894037 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.894246 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.894247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.894248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:44.894329 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.894331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.895100 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.895101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.896040 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.896043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.896763 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.896764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.896837 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.896842 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.896926 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.896966 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":43}
I20260923 15:11:44.897512 26694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6e95e32f4e4c0fa1050c691c3f68d1" candidate_uuid: "b6cb8506bbf047e2aaea582892fd24af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8a3fc7e86345adae498d355e6ad4af" is_pre_election: true
I20260923 15:11:44.897645 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.893818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37489 (local address 127.21.204.67:51271)
0923 15:11:44.893988 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.893999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:44.894074 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:44.894228 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:44.894229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:44.894231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:44.894330 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.894332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.895223 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.895224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.896402 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:44.896405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.897367 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.897368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:44.897447 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.897453 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 15:11:44.897514 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 15:11:44.897559 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":101,"threads_started":1}
I20260923 15:11:44.897652 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.893866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:51271 (local address 127.21.204.65:37489)
0923 15:11:44.893933 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.893935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.894066 (+ 131us) server_negotiation.cc:407] Connection header received
0923 15:11:44.894138 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.894139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.894149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.894221 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:44.894418 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.895125 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.896482 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.897129 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.897213 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.897224 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.897535 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.897583 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 15:11:44.897675 26694 raft_consensus.cc:2454] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cb8506bbf047e2aaea582892fd24af in term 0.
I20260923 15:11:44.898022 26760 leader_election.cc:302] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8a3fc7e86345adae498d355e6ad4af, b6cb8506bbf047e2aaea582892fd24af; no voters:
I20260923 15:11:44.898010 26561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6e95e32f4e4c0fa1050c691c3f68d1" candidate_uuid: "b6cb8506bbf047e2aaea582892fd24af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75718f341804d5eaefeaf28da35cccb" is_pre_election: true
I20260923 15:11:44.898152 26561 raft_consensus.cc:2454] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cb8506bbf047e2aaea582892fd24af in term 0.
I20260923 15:11:44.898201 26897 raft_consensus.cc:2781] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:44.898274 26897 raft_consensus.cc:482] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:44.898310 26897 raft_consensus.cc:3037] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:44.899065 26897 raft_consensus.cc:504] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.899206 26897 leader_election.cc:288] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [CANDIDATE]: Term 1 election: Requested vote from peers 6c8a3fc7e86345adae498d355e6ad4af (127.21.204.66:36595), e75718f341804d5eaefeaf28da35cccb (127.21.204.65:37489)
I20260923 15:11:44.899381 26694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6e95e32f4e4c0fa1050c691c3f68d1" candidate_uuid: "b6cb8506bbf047e2aaea582892fd24af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8a3fc7e86345adae498d355e6ad4af"
I20260923 15:11:44.899426 26561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6e95e32f4e4c0fa1050c691c3f68d1" candidate_uuid: "b6cb8506bbf047e2aaea582892fd24af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75718f341804d5eaefeaf28da35cccb"
I20260923 15:11:44.899457 26694 raft_consensus.cc:3037] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:44.899493 26561 raft_consensus.cc:3037] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:44.900321 26561 raft_consensus.cc:2454] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cb8506bbf047e2aaea582892fd24af in term 1.
I20260923 15:11:44.900326 26694 raft_consensus.cc:2454] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cb8506bbf047e2aaea582892fd24af in term 1.
I20260923 15:11:44.900576 26760 leader_election.cc:302] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6cb8506bbf047e2aaea582892fd24af, e75718f341804d5eaefeaf28da35cccb; no voters:
I20260923 15:11:44.900750 26897 raft_consensus.cc:2781] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:44.900892 26897 raft_consensus.cc:686] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 1 LEADER]: Becoming Leader. State: Replica: b6cb8506bbf047e2aaea582892fd24af, State: Running, Role: LEADER
I20260923 15:11:44.901006 26897 consensus_queue.cc:237] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } }
I20260923 15:11:44.901991 26416 catalog_manager.cc:5495] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cb8506bbf047e2aaea582892fd24af (127.21.204.67). New cstate: current_term: 1 leader_uuid: "b6cb8506bbf047e2aaea582892fd24af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cb8506bbf047e2aaea582892fd24af" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41355 } health_report { overall_health: HEALTHY } } }
W20260923 15:11:44.932121 26874 tablet.cc:2369] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:44.950874 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.948197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.21.204.67:41355)
0923 15:11:44.948271 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:44.948273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:44.948282 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:44.948317 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:44.948318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:44.948335 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:44.948399 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:44.948593 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.949062 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.949960 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:44.950152 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:44.950227 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:44.950516 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:44.950645 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:44.950720 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:44.950798 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 15:11:44.950817 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 15:11:44.953173 26561 raft_consensus.cc:1261] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P e75718f341804d5eaefeaf28da35cccb [term 1 FOLLOWER]: Refusing update from remote peer b6cb8506bbf047e2aaea582892fd24af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:44.953193 26694 raft_consensus.cc:1261] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 1 FOLLOWER]: Refusing update from remote peer b6cb8506bbf047e2aaea582892fd24af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:44.953572 26897 consensus_queue.cc:1035] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75718f341804d5eaefeaf28da35cccb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:44.953754 26900 consensus_queue.cc:1035] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8a3fc7e86345adae498d355e6ad4af" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:44.957206 26904 mvcc.cc:205] Tried to move back new op lower bound from 7332562145085063168 to 7332562144875347968. Current Snapshot: MvccSnapshot[applied={T|T < 7332562145085063168}]
I20260923 15:11:44.957326 26903 mvcc.cc:205] Tried to move back new op lower bound from 7332562145085063168 to 7332562144875347968. Current Snapshot: MvccSnapshot[applied={T|T < 7332562145085063168}]
I20260923 15:11:44.957412 26905 mvcc.cc:205] Tried to move back new op lower bound from 7332562145085063168 to 7332562144875347968. Current Snapshot: MvccSnapshot[applied={T|T < 7332562145085063168}]
I20260923 15:11:44.959759 26415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:44.959885 26415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:44.960788 26415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21c84eb659ce478694067ce93a1fee72: Sending DeleteTablet for 3 replicas of tablet 2c6e95e32f4e4c0fa1050c691c3f68d1
I20260923 15:11:44.961220 26807 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:44 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:50450
I20260923 15:11:44.961252 26541 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:44 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:49654
I20260923 15:11:44.961470 26913 tablet_replica.cc:330] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: stopping tablet replica
I20260923 15:11:44.961484 26674 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:44 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:59306
I20260923 15:11:44.961645 26913 raft_consensus.cc:2229] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:44.961804 26913 raft_consensus.cc:2258] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:44.961869 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 26479
I20260923 15:11:44.962744 26913 ts_tablet_manager.cc:1905] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:44.964392 26913 ts_tablet_manager.cc:1918] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:44.964473 26913 log.cc:1196] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/2c6e95e32f4e4c0fa1050c691c3f68d1
I20260923 15:11:44.964789 26913 ts_tablet_manager.cc:1939] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P b6cb8506bbf047e2aaea582892fd24af: Deleting consensus metadata
I20260923 15:11:44.965276 26404 catalog_manager.cc:4841] TS b6cb8506bbf047e2aaea582892fd24af (127.21.204.67:41355): tablet 2c6e95e32f4e4c0fa1050c691c3f68d1 (table test-table [id=9f3cce61ffef426a8d9e8cc885946ffa]) successfully deleted
W20260923 15:11:44.968779 26401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:37489 (error 108)
I20260923 15:11:44.968966 26915 tablet_replica.cc:330] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af: stopping tablet replica
I20260923 15:11:44.969053 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 26612
I20260923 15:11:44.969101 26915 raft_consensus.cc:2229] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:44.969218 26915 raft_consensus.cc:2258] T 2c6e95e32f4e4c0fa1050c691c3f68d1 P 6c8a3fc7e86345adae498d355e6ad4af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:44.970000 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.969772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37489 (local address 127.0.0.1:49668)
0923 15:11:44.969830 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.969958 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 15:11:44.970134 26401 catalog_manager.cc:4584] TS e75718f341804d5eaefeaf28da35cccb (127.21.204.65:37489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c6e95e32f4e4c0fa1050c691c3f68d1: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37489: connect: Connection refused (error 111)
I20260923 15:11:44.974982 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 26745
I20260923 15:11:44.975127 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:44.974982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:36595 (local address 127.0.0.1:59322)
0923 15:11:44.975049 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:44.975088 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.66:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 15:11:44.980731 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 26386
2026-09-23T15:11:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2212 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T15:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:45Z Disabled control of system clock
I20260923 15:11:45.012132 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42163
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 15:11:45.094949 26920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:45.095158 26920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:45.095180 26920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:45.096350 26920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:45.096392 26920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:45.096405 26920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:45.096418 26920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:45.096431 26920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:45.097879 26920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:42163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:45.098157 26920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:45.098469 26920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:45.103369 26925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.103370 26928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:45.103400 26920 server_base.cc:1034] running on GCE node
W20260923 15:11:45.103401 26926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:45.103797 26920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:45.104120 26920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:45.105351 26920 hybrid_clock.cc:648] HybridClock initialized: now 1790176305105334 us; error 33 us; skew 500 ppm
I20260923 15:11:45.110452 26920 webserver.cc:460] Webserver started at http://127.21.204.126:45735/ using document root <none> and password file <none>
I20260923 15:11:45.110872 26920 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:45.110975 26920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:45.111140 26920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:45.112389 26920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "aac9698ae0854f07b28f07cf1869d6a4"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "e56fbc5f34bf624eaf8760b6e17220a7"
server_key_iv: "3e1788615277faf9b3ce523e9d2d9301"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.112782 26920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "aac9698ae0854f07b28f07cf1869d6a4"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "e56fbc5f34bf624eaf8760b6e17220a7"
server_key_iv: "3e1788615277faf9b3ce523e9d2d9301"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.114260 26920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:45.115113 26934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.115352 26920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:45.115451 26920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "aac9698ae0854f07b28f07cf1869d6a4"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "e56fbc5f34bf624eaf8760b6e17220a7"
server_key_iv: "3e1788615277faf9b3ce523e9d2d9301"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.115513 26920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:45.118868 26920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:45.119093 26920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:45.119190 26920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:45.122994 26920 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:42163
I20260923 15:11:45.123061 26986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:42163 every 8 connection(s)
I20260923 15:11:45.123392 26920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:45.124300 26987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:45.127148 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 26920
I20260923 15:11:45.127233 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:45.127192 26987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4: Bootstrap starting.
I20260923 15:11:45.127420 22321 external_mini_cluster.cc:1428] Setting key cf4596751e95486485ad4a9ccb580a8d
I20260923 15:11:45.128094 26987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:45.128473 26987 log.cc:824] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:45.129288 26987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4: No bootstrap required, opened a new log
I20260923 15:11:45.130600 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.128152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.21.204.126:42163)
0923 15:11:45.128370 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.128381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.128419 (+ 38us) server_negotiation.cc:407] Connection header received
0923 15:11:45.128947 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.128957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.129012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.129178 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.130090 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.130093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.130094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.130099 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.130155 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.130170 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.130263 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.130354 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.130410 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260923 15:11:45.131557 26987 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9698ae0854f07b28f07cf1869d6a4" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42163 } }
I20260923 15:11:45.131695 26987 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:45.131717 26987 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac9698ae0854f07b28f07cf1869d6a4, State: Initialized, Role: FOLLOWER
I20260923 15:11:45.131848 26987 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9698ae0854f07b28f07cf1869d6a4" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42163 } }
I20260923 15:11:45.131943 26987 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:45.131989 26987 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:45.132040 26987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:45.132867 26987 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9698ae0854f07b28f07cf1869d6a4" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42163 } }
I20260923 15:11:45.132983 26987 leader_election.cc:302] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac9698ae0854f07b28f07cf1869d6a4; no voters:
I20260923 15:11:45.133143 26987 leader_election.cc:288] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:45.133167 26992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:45.133312 26992 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [term 1 LEADER]: Becoming Leader. State: Replica: aac9698ae0854f07b28f07cf1869d6a4, State: Running, Role: LEADER
I20260923 15:11:45.133424 26987 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:45.133404 26992 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9698ae0854f07b28f07cf1869d6a4" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42163 } }
I20260923 15:11:45.133832 26994 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac9698ae0854f07b28f07cf1869d6a4. Latest consensus state: current_term: 1 leader_uuid: "aac9698ae0854f07b28f07cf1869d6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9698ae0854f07b28f07cf1869d6a4" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42163 } } }
I20260923 15:11:45.133839 26993 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac9698ae0854f07b28f07cf1869d6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac9698ae0854f07b28f07cf1869d6a4" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42163 } } }
I20260923 15:11:45.133992 26994 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:45.134032 26993 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4 [sys.catalog]: This master's current role is: LEADER
W20260923 15:11:45.135385 27008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:11:45.135445 27008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:11:45.135500 27001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:45.135694 27001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:45.137199 27001 catalog_manager.cc:1347] Generated new cluster ID: 4e5239f6c6a04c01a4e4b57ca9b19b3a
I20260923 15:11:45.137262 27001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:45.152031 27001 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:45.152257 27001 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:45.156081 27001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4: Generated new TSK 0
I20260923 15:11:45.156288 27001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:11:45.162158 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42163
--builtin_ntp_servers=127.21.204.84:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 15:11:45.252276 27011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:45.252511 27011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:45.252547 27011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:45.253732 27011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:45.253800 27011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:45.253846 27011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:45.255234 27011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:45.255508 27011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:45.255810 27011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:45.262252 27017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.262295 27019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.262470 27016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:45.262847 27011 server_base.cc:1034] running on GCE node
I20260923 15:11:45.263104 27011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:45.263361 27011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:45.264575 27011 hybrid_clock.cc:648] HybridClock initialized: now 1790176305264557 us; error 32 us; skew 500 ppm
I20260923 15:11:45.266234 27011 webserver.cc:460] Webserver started at http://127.21.204.65:39027/ using document root <none> and password file <none>
I20260923 15:11:45.266659 27011 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:45.266765 27011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:45.266937 27011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:45.268098 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "1a5d22be2db341daa861e1b87dac9ffe"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "60453526e38b95b3471d662c823fc063"
server_key_iv: "ec72dae48c6eb2aa065dc611a58e7489"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.268476 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "1a5d22be2db341daa861e1b87dac9ffe"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "60453526e38b95b3471d662c823fc063"
server_key_iv: "ec72dae48c6eb2aa065dc611a58e7489"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.269922 27011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:45.270737 27025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.271010 27011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:45.271109 27011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "1a5d22be2db341daa861e1b87dac9ffe"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "60453526e38b95b3471d662c823fc063"
server_key_iv: "ec72dae48c6eb2aa065dc611a58e7489"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.271195 27011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:45.276064 27011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:45.276360 27011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:45.276506 27011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:45.276789 27011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:45.277196 27011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:45.277256 27011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.277293 27011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:45.277328 27011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.284155 27011 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:37021
I20260923 15:11:45.284255 27138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:37021 every 8 connection(s)
I20260923 15:11:45.284545 27011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:45.287837 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27011
I20260923 15:11:45.287950 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:45.288172 22321 external_mini_cluster.cc:1428] Setting key 4a6f1f0cc9a1bf996d374c06a815ea49
I20260923 15:11:45.288296 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.285407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:58933 (local address 127.21.204.126:42163)
0923 15:11:45.285456 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.285459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.285754 (+ 295us) server_negotiation.cc:407] Connection header received
0923 15:11:45.286495 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.286496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.286504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.286589 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.287978 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.287980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.287980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.287985 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.288002 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.288007 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.288069 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.288215 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.288231 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:11:45.288587 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.285346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42163 (local address 127.21.204.65:58933)
0923 15:11:45.285551 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.285618 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.286379 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.286687 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.286692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.286701 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.287785 (+ 1084us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.287785 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.287828 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.287862 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.288091 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.288102 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.288105 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.288225 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.288294 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260923 15:11:45.289264 27139 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42163
I20260923 15:11:45.289378 27139 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:45.289528 27139 heartbeater.cc:511] Master 127.21.204.126:42163 requested a full tablet report, sending...
I20260923 15:11:45.289654 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42163
--builtin_ntp_servers=127.21.204.84:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 15:11:45.291215 26951 ts_manager.cc:194] Registered new tserver with Master: 1a5d22be2db341daa861e1b87dac9ffe (127.21.204.65:37021)
W20260923 15:11:45.373680 27142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:45.373976 27142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:45.374007 27142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:45.375267 27142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:45.375305 27142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:45.375332 27142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:45.376777 27142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:45.376993 27142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:45.377247 27142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:45.382323 27147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.382436 27148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.382584 27150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:45.382824 27142 server_base.cc:1034] running on GCE node
I20260923 15:11:45.383101 27142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:45.383390 27142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:45.384547 27142 hybrid_clock.cc:648] HybridClock initialized: now 1790176305384535 us; error 37 us; skew 500 ppm
I20260923 15:11:45.386094 27142 webserver.cc:460] Webserver started at http://127.21.204.66:46587/ using document root <none> and password file <none>
I20260923 15:11:45.386370 27142 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:45.386421 27142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:45.386564 27142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:45.387676 27142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "d8109c5082b046a1992f109b626e2afc"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "4b631134ab3f040a25e38bb7332b30f3"
server_key_iv: "41c201ab051503afa3d43c003d3e1b49"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.388047 27142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "d8109c5082b046a1992f109b626e2afc"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "4b631134ab3f040a25e38bb7332b30f3"
server_key_iv: "41c201ab051503afa3d43c003d3e1b49"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.389487 27142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:45.390539 27156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.390827 27142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:11:45.390914 27142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "d8109c5082b046a1992f109b626e2afc"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "4b631134ab3f040a25e38bb7332b30f3"
server_key_iv: "41c201ab051503afa3d43c003d3e1b49"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.390980 27142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:45.394347 27142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:45.394629 27142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:45.394740 27142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:45.394908 27142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:45.395215 27142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:45.395246 27142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.395267 27142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:45.395304 27142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.400764 27142 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:33763
I20260923 15:11:45.400841 27269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:33763 every 8 connection(s)
I20260923 15:11:45.401160 27142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:45.404068 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.401950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:40085 (local address 127.21.204.126:42163)
0923 15:11:45.401997 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.401999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.402250 (+ 251us) server_negotiation.cc:407] Connection header received
0923 15:11:45.402703 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.402704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.402711 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.402794 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.403847 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.403848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.403849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.403853 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.403868 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.403872 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.403933 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.404002 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.404015 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:11:45.404311 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.401922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42163 (local address 127.21.204.66:40085)
0923 15:11:45.402114 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.402155 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.402618 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.402824 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.402827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.402836 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.403688 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.403689 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.403715 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.403742 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.403933 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.403935 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.403937 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.403986 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.404033 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260923 15:11:45.404865 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27142
I20260923 15:11:45.404983 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:45.405187 22321 external_mini_cluster.cc:1428] Setting key 61493b1e81152e200fc9a19d19011ad9
I20260923 15:11:45.405387 27270 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42163
I20260923 15:11:45.405489 27270 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:45.405638 27270 heartbeater.cc:511] Master 127.21.204.126:42163 requested a full tablet report, sending...
I20260923 15:11:45.406155 26951 ts_manager.cc:194] Registered new tserver with Master: d8109c5082b046a1992f109b626e2afc (127.21.204.66:33763)
I20260923 15:11:45.406279 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42163
--builtin_ntp_servers=127.21.204.84:44397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 15:11:45.488360 27273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:45.488863 27273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:45.488888 27273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:45.490147 27273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:45.490192 27273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:45.490218 27273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:45.491905 27273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:45.492187 27273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:45.492492 27273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:45.497635 27279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.497632 27278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:45.497718 27273 server_base.cc:1034] running on GCE node
W20260923 15:11:45.497632 27281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:45.499193 27273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:45.499449 27273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:45.500653 27273 hybrid_clock.cc:648] HybridClock initialized: now 1790176305500637 us; error 29 us; skew 500 ppm
I20260923 15:11:45.501959 27273 webserver.cc:460] Webserver started at http://127.21.204.67:38557/ using document root <none> and password file <none>
I20260923 15:11:45.502225 27273 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:45.502298 27273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:45.502436 27273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:45.503517 27273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "594ff264e43844daba3643e3939a0f74"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "92f4e594e87509cc4b14336479b73b63"
server_key_iv: "2fbb953df6e4f987d53ad5bc9961605c"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.503907 27273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "594ff264e43844daba3643e3939a0f74"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "92f4e594e87509cc4b14336479b73b63"
server_key_iv: "2fbb953df6e4f987d53ad5bc9961605c"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.505249 27273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:45.506050 27287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.506263 27273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:45.506366 27273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "594ff264e43844daba3643e3939a0f74"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "92f4e594e87509cc4b14336479b73b63"
server_key_iv: "2fbb953df6e4f987d53ad5bc9961605c"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.506449 27273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:45.509776 27273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:45.510090 27273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:45.510242 27273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:45.510448 27273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:45.510808 27273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:45.510860 27273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.510900 27273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:45.510927 27273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.517184 27273 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:43489
I20260923 15:11:45.517282 27400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:43489 every 8 connection(s)
I20260923 15:11:45.517616 27273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:45.521091 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.518484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:35171 (local address 127.21.204.126:42163)
0923 15:11:45.518605 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.518607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.518850 (+ 243us) server_negotiation.cc:407] Connection header received
0923 15:11:45.519412 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.519413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.519419 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.519476 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.520814 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.520816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.520816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.520821 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.520844 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.520848 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.520918 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.521003 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.521019 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 15:11:45.521095 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27273
I20260923 15:11:45.521227 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:45.521312 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.518437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42163 (local address 127.21.204.67:35171)
0923 15:11:45.518711 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.518748 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.519319 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.519549 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.519552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.519563 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.520609 (+ 1046us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.520610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.520638 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.520668 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.520925 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.520931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.520934 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.521004 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.521081 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260923 15:11:45.521414 22321 external_mini_cluster.cc:1428] Setting key b8decfbec25f23e6613e194e539d1149
I20260923 15:11:45.521795 27401 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42163
I20260923 15:11:45.521916 27401 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:45.522071 27401 heartbeater.cc:511] Master 127.21.204.126:42163 requested a full tablet report, sending...
I20260923 15:11:45.522481 26951 ts_manager.cc:194] Registered new tserver with Master: 594ff264e43844daba3643e3939a0f74 (127.21.204.67:43489)
I20260923 15:11:45.522737 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:11:45.524888 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.524004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.21.204.126:42163)
0923 15:11:45.524073 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.524074 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.524180 (+ 106us) server_negotiation.cc:407] Connection header received
0923 15:11:45.524239 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.524240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.524243 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.524282 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.524738 (+ 456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.524739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.524739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.524742 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.524754 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.524758 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.524801 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.524836 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.524845 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 15:11:45.526388 26951 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:45.527915 26951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:45.531957 27411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.530160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:33763 (local address 127.0.0.1:44512)
0923 15:11:45.530485 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.530493 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.530527 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.530821 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.530828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.530834 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.531266 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.531266 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.531282 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.531305 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.531545 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.531552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.531554 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.531620 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.531634 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":89,"threads_started":1}
I20260923 15:11:45.532156 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.530227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:43489 (local address 127.0.0.1:53134)
0923 15:11:45.530493 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.530503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.530540 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.531070 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.531071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.531074 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.531603 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.531604 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.531610 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.531623 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.531881 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.531882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.531883 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.531929 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.531941 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":66,"threads_started":1}
I20260923 15:11:45.532181 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.530316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.21.204.66:33763)
0923 15:11:45.530650 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.530653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.530661 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:45.530681 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.530683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.530689 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.530801 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.531385 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.531394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.531395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.531399 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.531431 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.531465 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.531554 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.532076 (+ 522us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.532092 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":83,"threads_started":1}
I20260923 15:11:45.532383 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.530142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37021 (local address 127.0.0.1:48358)
0923 15:11:45.530304 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.530342 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.530398 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.531420 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.531420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.531423 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.531937 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.531938 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.531978 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.531993 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.532270 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.532271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.532272 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.532307 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.532320 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260923 15:11:45.532542 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.530377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.21.204.67:43489)
0923 15:11:45.530560 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.530564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.530574 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:45.530593 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.530594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.530601 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.530724 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.531704 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.531741 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.531741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.531746 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.531778 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.531798 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.531852 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.531995 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.532011 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260923 15:11:45.532629 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.530380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.21.204.65:37021)
0923 15:11:45.531289 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.531293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.531303 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:45.531329 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.531331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.531339 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.531420 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.532116 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.532124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.532124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.532128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.532161 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.532184 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.532488 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.532516 (+ 28us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.532535 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":77,"threads_started":1}
I20260923 15:11:45.532876 27204 tablet_service.cc:1467] Processing CreateTablet for tablet 628b8b91c3974f118225be69943ed537 (DEFAULT_TABLE table=test-table [id=1387276426264b39a4b42ecaffd937b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:45.533176 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628b8b91c3974f118225be69943ed537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:45.533154 27335 tablet_service.cc:1467] Processing CreateTablet for tablet 628b8b91c3974f118225be69943ed537 (DEFAULT_TABLE table=test-table [id=1387276426264b39a4b42ecaffd937b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:45.533514 27335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628b8b91c3974f118225be69943ed537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:45.535787 27415 tablet_bootstrap.cc:492] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Bootstrap starting.
I20260923 15:11:45.536388 27073 tablet_service.cc:1467] Processing CreateTablet for tablet 628b8b91c3974f118225be69943ed537 (DEFAULT_TABLE table=test-table [id=1387276426264b39a4b42ecaffd937b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:45.536478 27415 tablet_bootstrap.cc:654] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:45.536685 27073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628b8b91c3974f118225be69943ed537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:45.536836 27415 log.cc:824] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:45.537220 27416 tablet_bootstrap.cc:492] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Bootstrap starting.
I20260923 15:11:45.537717 27415 tablet_bootstrap.cc:492] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: No bootstrap required, opened a new log
I20260923 15:11:45.537786 27416 tablet_bootstrap.cc:654] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:45.537787 27415 ts_tablet_manager.cc:1397] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:11:45.538080 27416 log.cc:824] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:45.538733 27416 tablet_bootstrap.cc:492] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: No bootstrap required, opened a new log
I20260923 15:11:45.538775 27416 ts_tablet_manager.cc:1397] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 15:11:45.539881 27419 tablet_bootstrap.cc:492] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe: Bootstrap starting.
I20260923 15:11:45.540407 27415 raft_consensus.cc:348] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.540522 27415 raft_consensus.cc:374] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:45.540542 27415 raft_consensus.cc:729] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8109c5082b046a1992f109b626e2afc, State: Initialized, Role: FOLLOWER
I20260923 15:11:45.540588 27419 tablet_bootstrap.cc:654] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:45.540663 27415 consensus_queue.cc:260] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.540892 27419 log.cc:824] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:45.540910 27415 ts_tablet_manager.cc:1428] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 15:11:45.540920 27270 heartbeater.cc:503] Master 127.21.204.126:42163 was elected leader, sending a full tablet report...
I20260923 15:11:45.540911 27416 raft_consensus.cc:348] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.541055 27416 raft_consensus.cc:374] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:45.541113 27416 raft_consensus.cc:729] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594ff264e43844daba3643e3939a0f74, State: Initialized, Role: FOLLOWER
I20260923 15:11:45.541244 27416 consensus_queue.cc:260] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.541493 27416 ts_tablet_manager.cc:1428] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 15:11:45.541615 27401 heartbeater.cc:503] Master 127.21.204.126:42163 was elected leader, sending a full tablet report...
I20260923 15:11:45.542021 27419 tablet_bootstrap.cc:492] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe: No bootstrap required, opened a new log
I20260923 15:11:45.542104 27419 ts_tablet_manager.cc:1397] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:11:45.544245 27419 raft_consensus.cc:348] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.544406 27419 raft_consensus.cc:374] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:45.544456 27419 raft_consensus.cc:729] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5d22be2db341daa861e1b87dac9ffe, State: Initialized, Role: FOLLOWER
I20260923 15:11:45.544566 27419 consensus_queue.cc:260] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.544867 27419 ts_tablet_manager.cc:1428] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:45.544934 27139 heartbeater.cc:503] Master 127.21.204.126:42163 was elected leader, sending a full tablet report...
I20260923 15:11:45.612311 27423 raft_consensus.cc:482] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:45.612455 27423 raft_consensus.cc:504] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.612934 27423 leader_election.cc:288] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8109c5082b046a1992f109b626e2afc (127.21.204.66:33763), 594ff264e43844daba3643e3939a0f74 (127.21.204.67:43489)
I20260923 15:11:45.615039 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.613255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:33763 (local address 127.21.204.65:42289)
0923 15:11:45.613317 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.613330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.613370 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.613585 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.613590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.613593 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.614425 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.614426 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.614436 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.614453 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.614813 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.614814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.614815 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.614886 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.614923 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":19}
I20260923 15:11:45.615065 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.613255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:43489 (local address 127.21.204.65:42641)
0923 15:11:45.613448 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.613463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.613501 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.614051 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.614053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.614070 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:45.614476 (+ 406us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:11:45.614477 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:45.614482 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:11:45.614492 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:45.614813 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:45.614814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:45.614815 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 15:11:45.614889 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 15:11:45.614916 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":14,"thread_start_us":117,"threads_started":1}
I20260923 15:11:45.615080 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.613350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:42641 (local address 127.21.204.67:43489)
0923 15:11:45.613401 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.613403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.613564 (+ 161us) server_negotiation.cc:407] Connection header received
0923 15:11:45.613737 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.613739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.613746 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.614021 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.614684 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.614685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.614685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.614690 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.614706 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.614710 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.614771 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.614992 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.615010 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:45.615080 27412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.613356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:42289 (local address 127.21.204.66:33763)
0923 15:11:45.613401 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.613403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.613414 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:11:45.613446 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.613449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.613461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.613563 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.614645 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.614647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.614647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.614653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.614673 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.614678 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.614764 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.614992 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.615010 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:11:45.615373 27355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628b8b91c3974f118225be69943ed537" candidate_uuid: "1a5d22be2db341daa861e1b87dac9ffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594ff264e43844daba3643e3939a0f74" is_pre_election: true
I20260923 15:11:45.615543 27355 raft_consensus.cc:2454] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a5d22be2db341daa861e1b87dac9ffe in term 0.
I20260923 15:11:45.615491 27224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628b8b91c3974f118225be69943ed537" candidate_uuid: "1a5d22be2db341daa861e1b87dac9ffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8109c5082b046a1992f109b626e2afc" is_pre_election: true
I20260923 15:11:45.615706 27224 raft_consensus.cc:2454] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a5d22be2db341daa861e1b87dac9ffe in term 0.
I20260923 15:11:45.615813 27027 leader_election.cc:302] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a5d22be2db341daa861e1b87dac9ffe, 594ff264e43844daba3643e3939a0f74; no voters:
I20260923 15:11:45.615978 27423 raft_consensus.cc:2781] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:45.616029 27423 raft_consensus.cc:482] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:45.616055 27423 raft_consensus.cc:3037] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:45.616782 27423 raft_consensus.cc:504] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.616941 27423 leader_election.cc:288] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [CANDIDATE]: Term 1 election: Requested vote from peers d8109c5082b046a1992f109b626e2afc (127.21.204.66:33763), 594ff264e43844daba3643e3939a0f74 (127.21.204.67:43489)
I20260923 15:11:45.617219 27224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628b8b91c3974f118225be69943ed537" candidate_uuid: "1a5d22be2db341daa861e1b87dac9ffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8109c5082b046a1992f109b626e2afc"
I20260923 15:11:45.617244 27355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628b8b91c3974f118225be69943ed537" candidate_uuid: "1a5d22be2db341daa861e1b87dac9ffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594ff264e43844daba3643e3939a0f74"
I20260923 15:11:45.617323 27355 raft_consensus.cc:3037] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:45.617311 27224 raft_consensus.cc:3037] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:45.618137 27355 raft_consensus.cc:2454] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a5d22be2db341daa861e1b87dac9ffe in term 1.
I20260923 15:11:45.618223 27224 raft_consensus.cc:2454] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a5d22be2db341daa861e1b87dac9ffe in term 1.
I20260923 15:11:45.618327 27027 leader_election.cc:302] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a5d22be2db341daa861e1b87dac9ffe, 594ff264e43844daba3643e3939a0f74; no voters:
I20260923 15:11:45.618419 27423 raft_consensus.cc:2781] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:45.618579 27423 raft_consensus.cc:686] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [term 1 LEADER]: Becoming Leader. State: Replica: 1a5d22be2db341daa861e1b87dac9ffe, State: Running, Role: LEADER
I20260923 15:11:45.618692 27423 consensus_queue.cc:237] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } }
I20260923 15:11:45.619678 26950 catalog_manager.cc:5495] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a5d22be2db341daa861e1b87dac9ffe (127.21.204.65). New cstate: current_term: 1 leader_uuid: "1a5d22be2db341daa861e1b87dac9ffe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a5d22be2db341daa861e1b87dac9ffe" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:45.629639 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.628641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.21.204.65:37021)
0923 15:11:45.628706 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.628709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.628781 (+ 72us) server_negotiation.cc:407] Connection header received
0923 15:11:45.628829 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.628830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.628841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.628891 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.629416 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.629418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.629418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.629422 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.629436 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.629440 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.629485 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.629526 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.629537 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 15:11:45.631381 27224 raft_consensus.cc:1261] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 1 FOLLOWER]: Refusing update from remote peer 1a5d22be2db341daa861e1b87dac9ffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:45.631417 27355 raft_consensus.cc:1261] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 1 FOLLOWER]: Refusing update from remote peer 1a5d22be2db341daa861e1b87dac9ffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:45.631839 27423 consensus_queue.cc:1035] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "594ff264e43844daba3643e3939a0f74" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:45.632036 27425 consensus_queue.cc:1035] T 628b8b91c3974f118225be69943ed537 P 1a5d22be2db341daa861e1b87dac9ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8109c5082b046a1992f109b626e2afc" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 33763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:45.634639 27428 mvcc.cc:205] Tried to move back new op lower bound from 7332562147863576576 to 7332562147814875136. Current Snapshot: MvccSnapshot[applied={T|T < 7332562147863576576}]
I20260923 15:11:45.635468 27430 mvcc.cc:205] Tried to move back new op lower bound from 7332562147863576576 to 7332562147814875136. Current Snapshot: MvccSnapshot[applied={T|T < 7332562147863576576}]
I20260923 15:11:45.637166 26951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:45.637260 26951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:45.638024 26951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aac9698ae0854f07b28f07cf1869d6a4: Sending DeleteTablet for 3 replicas of tablet 628b8b91c3974f118225be69943ed537
I20260923 15:11:45.638342 27073 tablet_service.cc:1514] Processing DeleteTablet for tablet 628b8b91c3974f118225be69943ed537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:45 UTC) from {username='slave'} at 127.0.0.1:48358
I20260923 15:11:45.638440 27335 tablet_service.cc:1514] Processing DeleteTablet for tablet 628b8b91c3974f118225be69943ed537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:45 UTC) from {username='slave'} at 127.0.0.1:53134
I20260923 15:11:45.638459 27204 tablet_service.cc:1514] Processing DeleteTablet for tablet 628b8b91c3974f118225be69943ed537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:45 UTC) from {username='slave'} at 127.0.0.1:44512
I20260923 15:11:45.638605 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27011
I20260923 15:11:45.638651 27439 tablet_replica.cc:330] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: stopping tablet replica
I20260923 15:11:45.638748 27439 raft_consensus.cc:2229] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:45.638842 27439 raft_consensus.cc:2258] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:45.639272 27439 ts_tablet_manager.cc:1905] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:45.640916 27439 ts_tablet_manager.cc:1918] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:45.641021 27439 log.cc:1196] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/628b8b91c3974f118225be69943ed537
I20260923 15:11:45.641299 27439 ts_tablet_manager.cc:1939] T 628b8b91c3974f118225be69943ed537 P 594ff264e43844daba3643e3939a0f74: Deleting consensus metadata
I20260923 15:11:45.641707 26936 catalog_manager.cc:4841] TS 594ff264e43844daba3643e3939a0f74 (127.21.204.67:43489): tablet 628b8b91c3974f118225be69943ed537 (table test-table [id=1387276426264b39a4b42ecaffd937b3]) successfully deleted
I20260923 15:11:45.641993 27440 tablet_replica.cc:330] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: stopping tablet replica
I20260923 15:11:45.642094 27440 raft_consensus.cc:2229] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:45.642201 27440 raft_consensus.cc:2258] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:45.642562 27440 ts_tablet_manager.cc:1905] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:45.643939 27440 ts_tablet_manager.cc:1918] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:45.643998 27440 log.cc:1196] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/628b8b91c3974f118225be69943ed537
I20260923 15:11:45.644282 27440 ts_tablet_manager.cc:1939] T 628b8b91c3974f118225be69943ed537 P d8109c5082b046a1992f109b626e2afc: Deleting consensus metadata
I20260923 15:11:45.644644 26937 catalog_manager.cc:4841] TS d8109c5082b046a1992f109b626e2afc (127.21.204.66:33763): tablet 628b8b91c3974f118225be69943ed537 (table test-table [id=1387276426264b39a4b42ecaffd937b3]) successfully deleted
W20260923 15:11:45.645454 26938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:37021 (error 108)
I20260923 15:11:45.645622 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27142
I20260923 15:11:45.646476 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.646323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37021 (local address 127.0.0.1:48384)
0923 15:11:45.646386 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.646433 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 15:11:45.646598 26938 catalog_manager.cc:4584] TS 1a5d22be2db341daa861e1b87dac9ffe (127.21.204.65:37021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 628b8b91c3974f118225be69943ed537: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37021: connect: Connection refused (error 111)
I20260923 15:11:45.653256 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27273
I20260923 15:11:45.657967 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 26920
2026-09-23T15:11:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (676 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 15:11:45.678846 22321 mini_oidc.cc:152] Starting JWKS server
I20260923 15:11:45.679934 22321 webserver.cc:460] Webserver started at https://127.0.0.1:41893/ using document root <none> and password file <none>
I20260923 15:11:45.680066 22321 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 15:11:45.680261 22321 webserver.cc:460] Webserver started at http://127.0.0.1:44807/ using document root <none> and password file <none>
2026-09-23T15:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:45Z Disabled control of system clock
I20260923 15:11:45.688818 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41445
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:41445
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41893/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/testchainca.cert with env {}
W20260923 15:11:45.770875 27447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:45.771095 27447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:45.771118 27447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:45.772315 27447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:45.772372 27447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:45.772394 27447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:45.772413 27447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:45.772429 27447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 15:11:45.772449 27447 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 15:11:45.772469 27447 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41893/jwks.json
I20260923 15:11:45.774459 27447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:41445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41893/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:45.774658 27447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:45.774814 27447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:45.779909 27452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.779923 27455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:45.779932 27453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:45.780186 27447 server_base.cc:1034] running on GCE node
I20260923 15:11:45.780486 27447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:45.780786 27447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:45.782032 27447 hybrid_clock.cc:648] HybridClock initialized: now 1790176305782020 us; error 45 us; skew 500 ppm
I20260923 15:11:45.783371 27447 webserver.cc:460] Webserver started at http://127.21.204.126:42723/ using document root <none> and password file <none>
I20260923 15:11:45.783615 27447 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:45.783725 27447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:45.783879 27447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:45.784942 27447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "5739d767376c4202adb76150c1aa5032"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "8b12c76ca4a549e242164fe847af876d"
server_key_iv: "301cae1d47425fd1339a332b862b5899"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.785310 27447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "5739d767376c4202adb76150c1aa5032"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "8b12c76ca4a549e242164fe847af876d"
server_key_iv: "301cae1d47425fd1339a332b862b5899"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.786882 27447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:45.787617 27461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:45.787823 27447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:45.787933 27447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "5739d767376c4202adb76150c1aa5032"
format_stamp: "Formatted at 2026-09-23 15:11:45 on dist-test-slave-t5lt"
server_key: "8b12c76ca4a549e242164fe847af876d"
server_key_iv: "301cae1d47425fd1339a332b862b5899"
server_key_version: "encryptionkey@0"
I20260923 15:11:45.788012 27447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:45.804561 27447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:45.804941 27447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:45.805121 27447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:45.809356 27447 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:41445
I20260923 15:11:45.809415 27513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:41445 every 8 connection(s)
I20260923 15:11:45.809779 27447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:45.810904 27514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:45.813223 27514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032: Bootstrap starting.
I20260923 15:11:45.813678 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27447
I20260923 15:11:45.813809 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:45.813977 27514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:45.814018 22321 external_mini_cluster.cc:1428] Setting key a138ed468e8f63c8683c65c26d85ad47
I20260923 15:11:45.814280 27514 log.cc:824] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:45.815030 27514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032: No bootstrap required, opened a new log
I20260923 15:11:45.817057 27514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5739d767376c4202adb76150c1aa5032" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41445 } }
I20260923 15:11:45.817235 27514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:45.817284 27514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5739d767376c4202adb76150c1aa5032, State: Initialized, Role: FOLLOWER
I20260923 15:11:45.817453 27514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5739d767376c4202adb76150c1aa5032" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41445 } }
I20260923 15:11:45.817539 27514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:45.817579 27514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:45.817620 27514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:45.818369 27514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5739d767376c4202adb76150c1aa5032" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41445 } }
I20260923 15:11:45.818468 27514 leader_election.cc:302] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5739d767376c4202adb76150c1aa5032; no voters:
I20260923 15:11:45.818662 27514 leader_election.cc:288] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:45.818764 27519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:45.818946 27514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:45.819083 27519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [term 1 LEADER]: Becoming Leader. State: Replica: 5739d767376c4202adb76150c1aa5032, State: Running, Role: LEADER
I20260923 15:11:45.819180 27519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5739d767376c4202adb76150c1aa5032" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41445 } }
I20260923 15:11:45.819491 27520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5739d767376c4202adb76150c1aa5032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5739d767376c4202adb76150c1aa5032" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41445 } } }
I20260923 15:11:45.819577 27520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:45.819870 27521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5739d767376c4202adb76150c1aa5032. Latest consensus state: current_term: 1 leader_uuid: "5739d767376c4202adb76150c1aa5032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5739d767376c4202adb76150c1aa5032" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41445 } } }
I20260923 15:11:45.819989 27521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:45.819901 27528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:45.820107 27518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.814512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.21.204.126:41445)
0923 15:11:45.814723 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.814731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.814768 (+ 37us) server_negotiation.cc:407] Connection header received
0923 15:11:45.815265 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.815272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.815332 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.815486 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:45.815721 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:45.816585 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:45.817424 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:45.818080 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:45.818221 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:45.819015 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:45.819020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:45.819021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:45.819234 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:45.819295 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:45.819310 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:45.819377 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:45.819679 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 15:11:45.819774 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260923 15:11:45.820880 27528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:45.821157 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:11:45.822876 27528 catalog_manager.cc:1347] Generated new cluster ID: 8883a3d80494452289cd4affb68c5066
I20260923 15:11:45.822956 27528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:45.833743 27528 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:45.834321 27528 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:45.840373 27528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5739d767376c4202adb76150c1aa5032: Generated new TSK 0
I20260923 15:11:45.840595 27528 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 15:11:45.872301 27544 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 15:11:45.872409 27544 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 15:11:45.872479 27518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:45.846621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.21.204.126:41445)
0923 15:11:45.846679 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:45.846681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:45.846790 (+ 109us) server_negotiation.cc:407] Connection header received
0923 15:11:45.846908 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:45.846908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:45.846922 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:45.846984 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0923 15:11:45.847156 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:45.847656 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:45.848478 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:45.848697 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:45.848781 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:45.849017 (+ 236us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 15:11:45.872205 (+ 23188us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 15:11:45.872400 (+ 195us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51412: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260923 15:11:45.872558 27518 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51412: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 15:11:45.873124 27543 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:45.846533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41445 (local address 127.0.0.1:51412)
0923 15:11:45.846703 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:45.846713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:45.846801 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:45.847003 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:45.847004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:45.847006 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 15:11:45.847064 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:45.847068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:45.847762 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:45.847763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:45.848382 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:45.848384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:45.848791 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:45.848792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:45.848849 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:45.848877 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 15:11:45.872362 (+ 23485us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 15:11:45.872877 (+ 515us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.204.126:41445: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.204.126:41445: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 15:11:45.873647 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27447
2026-09-23T15:11:45Z chronyd exiting
I20260923 15:11:45.910106 22321 test_util.cc:175] -----------------------------------------------
I20260923 15:11:45.910252 22321 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790176278407460-22321-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 15:11:45.911366 22321 mini_oidc.cc:152] Starting JWKS server
I20260923 15:11:45.912441 22321 webserver.cc:460] Webserver started at https://127.0.0.1:40223/ using document root <none> and password file <none>
I20260923 15:11:45.912501 22321 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 15:11:45.912663 22321 webserver.cc:460] Webserver started at http://127.0.0.1:35693/ using document root <none> and password file <none>
2026-09-23T15:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:45Z Disabled control of system clock
I20260923 15:11:45.922714 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35617
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:35617
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40223/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/testchainca.cert with env {}
W20260923 15:11:46.007277 27550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:46.007486 27550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:46.007509 27550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:46.008632 27550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:46.008669 27550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:46.008682 27550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:46.008694 27550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:46.008708 27550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 15:11:46.008719 27550 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 15:11:46.008730 27550 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40223/jwks.json
I20260923 15:11:46.010241 27550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:35617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40223/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:46.010435 27550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:46.010596 27550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:11:46.015872 27550 server_base.cc:1034] running on GCE node
W20260923 15:11:46.015798 27555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:46.015784 27556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:46.015780 27558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:46.016227 27550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:46.016582 27550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:46.017758 27550 hybrid_clock.cc:648] HybridClock initialized: now 1790176306017747 us; error 28 us; skew 500 ppm
I20260923 15:11:46.019052 27550 webserver.cc:460] Webserver started at http://127.21.204.126:35663/ using document root <none> and password file <none>
I20260923 15:11:46.019332 27550 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:46.019403 27550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:46.019544 27550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:46.020625 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "60dbb95be8444c479698ceaaa29fef89"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "2bb5677734620aff4c8288afc68fc83a"
server_key_iv: "6ec8d1ae0ad169c5f098559c9cead932"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.020984 27550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "60dbb95be8444c479698ceaaa29fef89"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "2bb5677734620aff4c8288afc68fc83a"
server_key_iv: "6ec8d1ae0ad169c5f098559c9cead932"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.022454 27550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:46.023304 27564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:46.023531 27550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:46.023671 27550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "60dbb95be8444c479698ceaaa29fef89"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "2bb5677734620aff4c8288afc68fc83a"
server_key_iv: "6ec8d1ae0ad169c5f098559c9cead932"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.023751 27550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:46.037165 27550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:46.037499 27550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:46.037667 27550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:46.041501 27550 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:35617
I20260923 15:11:46.041632 27616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:35617 every 8 connection(s)
I20260923 15:11:46.041982 27550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:46.042750 27617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:46.045485 27617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89: Bootstrap starting.
I20260923 15:11:46.046149 27617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:46.046438 27617 log.cc:824] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:46.047127 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27550
I20260923 15:11:46.047248 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:46.047303 27617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89: No bootstrap required, opened a new log
I20260923 15:11:46.047459 22321 external_mini_cluster.cc:1428] Setting key 019f4d5d1e4820d566a8a285eca5e210
I20260923 15:11:46.049177 27617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dbb95be8444c479698ceaaa29fef89" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35617 } }
I20260923 15:11:46.049337 27617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:46.049362 27617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dbb95be8444c479698ceaaa29fef89, State: Initialized, Role: FOLLOWER
I20260923 15:11:46.049491 27617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dbb95be8444c479698ceaaa29fef89" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35617 } }
I20260923 15:11:46.049593 27617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:46.049638 27617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:46.049697 27617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:46.050498 27617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dbb95be8444c479698ceaaa29fef89" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35617 } }
I20260923 15:11:46.050616 27617 leader_election.cc:302] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60dbb95be8444c479698ceaaa29fef89; no voters:
I20260923 15:11:46.050772 27617 leader_election.cc:288] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:46.050848 27622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:46.050998 27622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [term 1 LEADER]: Becoming Leader. State: Replica: 60dbb95be8444c479698ceaaa29fef89, State: Running, Role: LEADER
I20260923 15:11:46.051028 27617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:46.051090 27622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dbb95be8444c479698ceaaa29fef89" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35617 } }
I20260923 15:11:46.051426 27622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60dbb95be8444c479698ceaaa29fef89. Latest consensus state: current_term: 1 leader_uuid: "60dbb95be8444c479698ceaaa29fef89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dbb95be8444c479698ceaaa29fef89" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35617 } } }
I20260923 15:11:46.051527 27622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:46.051442 27623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60dbb95be8444c479698ceaaa29fef89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60dbb95be8444c479698ceaaa29fef89" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35617 } } }
I20260923 15:11:46.051677 27623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89 [sys.catalog]: This master's current role is: LEADER
W20260923 15:11:46.052878 27638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:11:46.053037 27638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:11:46.053126 27630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:46.053292 27630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:46.055610 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:46.048018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.21.204.126:35617)
0923 15:11:46.048250 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:46.048257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:46.048282 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:11:46.049193 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:46.049201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:46.049309 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:46.049484 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:46.049729 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.051558 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.053124 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.053835 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.054008 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.054901 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:46.054905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:46.054906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:46.055085 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:46.055117 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:46.055129 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:46.055212 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:46.055289 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 15:11:46.055371 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260923 15:11:46.055636 27630 catalog_manager.cc:1347] Generated new cluster ID: cfba2821674a4274be7e091bb7d95471
I20260923 15:11:46.055671 27630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:46.057363 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:11:46.066324 27630 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:46.066851 27630 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:46.072369 27630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60dbb95be8444c479698ceaaa29fef89: Generated new TSK 0
I20260923 15:11:46.072542 27630 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 15:11:46.092533 27647 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 15:11:46.092634 27647 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260923 15:11:46.093047 27646 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:46.077641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:35617 (local address 127.0.0.1:51426)
0923 15:11:46.077929 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:46.077940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:46.078007 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:46.078169 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:46.078170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:46.078172 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 15:11:46.078223 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:46.078225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.079414 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.079415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:46.080038 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:46.080040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.080534 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.080535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:46.080575 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.080579 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 15:11:46.092874 (+ 12295us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 15:11:46.092980 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.204.126:35617: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
I20260923 15:11:46.093031 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:46.077806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.21.204.126:35617)
0923 15:11:46.077899 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:46.077902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:46.078013 (+ 111us) server_negotiation.cc:407] Connection header received
0923 15:11:46.078057 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:46.078058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:46.078071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:46.078151 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0923 15:11:46.078291 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.079047 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.080129 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.080452 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.080524 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.080651 (+ 127us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 15:11:46.092756 (+ 12105us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 15:11:46.092940 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 15:11:46.093114 27621 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260923 15:11:46.093477 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27550
2026-09-23T15:11:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 15:11:46.131816 22321 mini_oidc.cc:152] Starting JWKS server
I20260923 15:11:46.132516 22321 webserver.cc:460] Webserver started at https://127.0.0.1:45229/ using document root <none> and password file <none>
I20260923 15:11:46.132589 22321 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 15:11:46.132761 22321 webserver.cc:460] Webserver started at http://127.0.0.1:37837/ using document root <none> and password file <none>
2026-09-23T15:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:46Z Disabled control of system clock
I20260923 15:11:46.140331 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35921
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:35055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:35921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45229/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 15:11:46.220963 27653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:46.221175 27653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:46.221199 27653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:46.222481 27653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:46.222522 27653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:46.222535 27653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:46.222548 27653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:46.222560 27653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 15:11:46.222613 27653 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 15:11:46.222630 27653 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45229/jwks.json
I20260923 15:11:46.224041 27653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:35921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45229/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:46.224319 27653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:46.224642 27653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:46.229528 27658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:46.230090 27661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:46.230139 27659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:46.230481 27653 server_base.cc:1034] running on GCE node
I20260923 15:11:46.230707 27653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:46.230957 27653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:46.232136 27653 hybrid_clock.cc:648] HybridClock initialized: now 1790176306232103 us; error 50 us; skew 500 ppm
I20260923 15:11:46.233552 27653 webserver.cc:460] Webserver started at http://127.21.204.126:35743/ using document root <none> and password file <none>
I20260923 15:11:46.233762 27653 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:46.233805 27653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:46.233984 27653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:46.235118 27653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "10e049e4c7a74f19b35a5e9ee545d2ab"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "27a9d956e99440baa14537e7fa20ba80"
server_key_iv: "5d650bb8015666d0d5d027f47322c6f2"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.235448 27653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "10e049e4c7a74f19b35a5e9ee545d2ab"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "27a9d956e99440baa14537e7fa20ba80"
server_key_iv: "5d650bb8015666d0d5d027f47322c6f2"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.236829 27653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:46.237659 27667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:46.237967 27653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:46.238075 27653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "10e049e4c7a74f19b35a5e9ee545d2ab"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "27a9d956e99440baa14537e7fa20ba80"
server_key_iv: "5d650bb8015666d0d5d027f47322c6f2"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.238175 27653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:46.245309 27653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:46.245611 27653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:46.245764 27653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:46.249527 27653 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:35921
I20260923 15:11:46.249591 27719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:35921 every 8 connection(s)
I20260923 15:11:46.250049 27653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:46.250749 27720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:46.253443 27720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab: Bootstrap starting.
I20260923 15:11:46.254122 27720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:46.254411 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27653
I20260923 15:11:46.254418 27720 log.cc:824] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:46.254539 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:46.254786 22321 external_mini_cluster.cc:1428] Setting key 0d83f37cc3be6a908b6f1dcdd00a90aa
I20260923 15:11:46.255960 27720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab: No bootstrap required, opened a new log
I20260923 15:11:46.258215 27720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35921 } }
I20260923 15:11:46.258383 27720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:46.258414 27720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e049e4c7a74f19b35a5e9ee545d2ab, State: Initialized, Role: FOLLOWER
I20260923 15:11:46.258546 27720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35921 } }
I20260923 15:11:46.258620 27720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:46.258648 27720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:46.258687 27720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:46.259284 27720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35921 } }
I20260923 15:11:46.259394 27720 leader_election.cc:302] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10e049e4c7a74f19b35a5e9ee545d2ab; no voters:
I20260923 15:11:46.259903 27720 leader_election.cc:288] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:46.259933 27725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:46.260169 27725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [term 1 LEADER]: Becoming Leader. State: Replica: 10e049e4c7a74f19b35a5e9ee545d2ab, State: Running, Role: LEADER
I20260923 15:11:46.260219 27720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:46.260267 27725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35921 } }
I20260923 15:11:46.260596 27726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35921 } } }
I20260923 15:11:46.260697 27726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:46.260932 27727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e049e4c7a74f19b35a5e9ee545d2ab. Latest consensus state: current_term: 1 leader_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e049e4c7a74f19b35a5e9ee545d2ab" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35921 } } }
I20260923 15:11:46.260982 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:46.255318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.21.204.126:35921)
0923 15:11:46.255749 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:46.255758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:46.255793 (+ 35us) server_negotiation.cc:407] Connection header received
0923 15:11:46.256311 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:46.256319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:46.256379 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:46.256491 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:46.256755 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.257644 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.258773 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.259187 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.259278 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.260009 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:46.260014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:46.260014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:46.260231 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:46.260274 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:46.260290 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:46.260653 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:46.260687 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 15:11:46.260769 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":76,"threads_started":1}
I20260923 15:11:46.261066 27727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:46.261404 27732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:46.261940 27732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:46.262086 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:11:46.263473 27732 catalog_manager.cc:1347] Generated new cluster ID: 9f87cb4462cb477dbdbe4800b150b7a5
I20260923 15:11:46.263525 27732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:46.270396 27732 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:46.270910 27732 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:46.275794 27732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10e049e4c7a74f19b35a5e9ee545d2ab: Generated new TSK 0
I20260923 15:11:46.275995 27732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:11:46.286568 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:46.282524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.21.204.126:35921)
0923 15:11:46.282582 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:46.282584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:46.282683 (+ 99us) server_negotiation.cc:407] Connection header received
0923 15:11:46.282745 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:46.282746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:46.282758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:46.282845 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0923 15:11:46.283028 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.283810 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.284646 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.284972 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.285045 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.285267 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 15:11:46.286351 (+ 1084us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 15:11:46.286488 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":28}
W20260923 15:11:46.286626 27724 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 15:11:46.288483 27749 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:46.282343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:35921 (local address 127.0.0.1:39768)
0923 15:11:46.282593 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:46.282607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:46.282685 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:46.282892 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:46.282893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:46.282895 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 15:11:46.282949 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:46.282950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.283950 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.283952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:46.284558 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:46.284560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.285087 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.285095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:46.285151 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.285156 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 15:11:46.286459 (+ 1303us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 15:11:46.288387 (+ 1928us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.204.126:35921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1740,"thread_start_us":111,"threads_started":1}
I20260923 15:11:46.288704 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27653
E20260923 15:11:46.298110 27750 webserver.cc:533] Webserver: sslize failed:
E20260923 15:11:46.298177 27750 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T15:11:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (194 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T15:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:46Z Disabled control of system clock
I20260923 15:11:46.334156 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42095
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:42095
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/testchainca.cert with env {}
W20260923 15:11:46.416699 27754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:46.416952 27754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:46.416993 27754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:46.418290 27754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:46.418359 27754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:46.418382 27754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:46.418409 27754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:46.418442 27754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 15:11:46.418465 27754 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 15:11:46.418489 27754 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 15:11:46.420007 27754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:42095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:46.420305 27754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:46.420531 27754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:46.425586 27762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:46.425585 27759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:46.425617 27760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:46.425973 27754 server_base.cc:1034] running on GCE node
I20260923 15:11:46.426162 27754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:46.426411 27754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:46.427640 27754 hybrid_clock.cc:648] HybridClock initialized: now 1790176306427627 us; error 29 us; skew 500 ppm
I20260923 15:11:46.429035 27754 webserver.cc:460] Webserver started at http://127.21.204.126:42271/ using document root <none> and password file <none>
I20260923 15:11:46.429345 27754 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:46.429397 27754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:46.429533 27754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:46.430670 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "f308439ad8e646c6837a13f9e69bb149"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "0be4f2c6806b52e35a5d228b85f46ee8"
server_key_iv: "56e7a010c8a8685354fef1d530219fe1"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.431030 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "f308439ad8e646c6837a13f9e69bb149"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "0be4f2c6806b52e35a5d228b85f46ee8"
server_key_iv: "56e7a010c8a8685354fef1d530219fe1"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.432359 27754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:11:46.433122 27768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:46.433360 27754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:46.433462 27754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "f308439ad8e646c6837a13f9e69bb149"
format_stamp: "Formatted at 2026-09-23 15:11:46 on dist-test-slave-t5lt"
server_key: "0be4f2c6806b52e35a5d228b85f46ee8"
server_key_iv: "56e7a010c8a8685354fef1d530219fe1"
server_key_version: "encryptionkey@0"
I20260923 15:11:46.433516 27754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:46.448918 27754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:46.449223 27754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:46.449359 27754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:46.453502 27754 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:42095
I20260923 15:11:46.453536 27820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:42095 every 8 connection(s)
I20260923 15:11:46.453982 27754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:46.455015 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:46.457425 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149: Bootstrap starting.
I20260923 15:11:46.458119 27821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:46.458426 27821 log.cc:824] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:46.458739 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27754
I20260923 15:11:46.458855 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:46.459090 22321 external_mini_cluster.cc:1428] Setting key 21ced8ecaa4178c9707708a1afde44c2
I20260923 15:11:46.459292 27821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149: No bootstrap required, opened a new log
I20260923 15:11:46.461107 27821 raft_consensus.cc:348] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f308439ad8e646c6837a13f9e69bb149" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42095 } }
I20260923 15:11:46.461241 27821 raft_consensus.cc:374] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:46.461263 27821 raft_consensus.cc:729] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f308439ad8e646c6837a13f9e69bb149, State: Initialized, Role: FOLLOWER
I20260923 15:11:46.461401 27821 consensus_queue.cc:260] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f308439ad8e646c6837a13f9e69bb149" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42095 } }
I20260923 15:11:46.461505 27821 raft_consensus.cc:388] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:46.461550 27821 raft_consensus.cc:482] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:46.461609 27821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:46.462378 27821 raft_consensus.cc:504] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f308439ad8e646c6837a13f9e69bb149" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42095 } }
I20260923 15:11:46.462491 27821 leader_election.cc:302] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f308439ad8e646c6837a13f9e69bb149; no voters:
I20260923 15:11:46.462637 27821 leader_election.cc:288] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:46.462739 27826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:46.462875 27821 sys_catalog.cc:564] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:46.462879 27826 raft_consensus.cc:686] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [term 1 LEADER]: Becoming Leader. State: Replica: f308439ad8e646c6837a13f9e69bb149, State: Running, Role: LEADER
I20260923 15:11:46.462996 27826 consensus_queue.cc:237] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f308439ad8e646c6837a13f9e69bb149" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42095 } }
I20260923 15:11:46.463362 27827 sys_catalog.cc:455] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f308439ad8e646c6837a13f9e69bb149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f308439ad8e646c6837a13f9e69bb149" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42095 } } }
I20260923 15:11:46.463450 27827 sys_catalog.cc:458] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:46.463629 27828 sys_catalog.cc:455] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f308439ad8e646c6837a13f9e69bb149. Latest consensus state: current_term: 1 leader_uuid: "f308439ad8e646c6837a13f9e69bb149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f308439ad8e646c6837a13f9e69bb149" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42095 } } }
I20260923 15:11:46.463712 27828 sys_catalog.cc:458] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:46.463846 27834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:46.464342 27834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:46.465900 27834 catalog_manager.cc:1347] Generated new cluster ID: 0d1ff76ab64048f291a15a28c3c20d43
I20260923 15:11:46.465946 27834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:46.467065 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:46.459649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.21.204.126:42095)
0923 15:11:46.459899 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:46.459907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:46.459934 (+ 27us) server_negotiation.cc:407] Connection header received
0923 15:11:46.461123 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:46.461131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:46.461199 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:46.461402 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:46.461640 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.463412 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.464679 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.465275 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.465392 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.466353 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:46.466357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:46.466357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:11:46.466564 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:46.466604 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:11:46.466617 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:46.466687 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:46.466797 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 15:11:46.466868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":106,"threads_started":1}
I20260923 15:11:46.467976 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:11:46.476181 27834 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:46.476678 27834 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:46.488863 27834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f308439ad8e646c6837a13f9e69bb149: Generated new TSK 0
I20260923 15:11:46.489143 27834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:11:46.584975 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:46.501184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.21.204.126:42095)
0923 15:11:46.501262 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:46.501264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:46.501380 (+ 116us) server_negotiation.cc:407] Connection header received
0923 15:11:46.501463 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:46.501464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:46.501477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:46.501547 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0923 15:11:46.501703 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.502300 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.503148 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.503355 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.503444 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.503644 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 15:11:46.584645 (+ 81001us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 15:11:46.584876 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40126: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 15:11:46.585064 27825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40126: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 15:11:46.585075 27850 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:46.501121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42095 (local address 127.0.0.1:40126)
0923 15:11:46.501307 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:46.501317 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:46.501398 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:46.501558 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:46.501559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:46.501561 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 15:11:46.501622 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:46.501624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.502396 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.502397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:46.503026 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:46.503029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:46.503454 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:46.503456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:46.503515 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:46.503521 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 15:11:46.584823 (+ 81302us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 15:11:46.584962 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.204.126:42095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260923 15:11:46.585623 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27754
2026-09-23T15:11:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (277 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:46 dist-test-slave-t5lt krb5kdc[27862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:46 dist-test-slave-t5lt krb5kdc[27862](info): set up 2 sockets
Sep 23 15:11:46 dist-test-slave-t5lt krb5kdc[27862](info): commencing operation
krb5kdc: starting...
W20260923 15:11:47.811357 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:47 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:47.933501 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41213
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:41213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:48.016579 27875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:48.016803 27875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:48.016827 27875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:48.017990 27875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:48.018025 27875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:48.018039 27875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:48.018052 27875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:48.018064 27875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:48.019457 27875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:41213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:48.019650 27875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:48.019822 27875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:48.024756 27883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:48.024798 27875 server_base.cc:1034] running on GCE node
W20260923 15:11:48.024768 27881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:48.024775 27880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:48.025271 27875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:48.025555 27875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:48.026717 27875 hybrid_clock.cc:648] HybridClock initialized: now 1790176308026696 us; error 42 us; skew 500 ppm
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.029274 27875 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:11:48.029701 27875 webserver.cc:460] Webserver started at http://127.21.204.126:39055/ using document root <none> and password file <none>
I20260923 15:11:48.029959 27875 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:48.030027 27875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:48.030184 27875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:48.031215 27875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "8c9a9dc6a89f404cb22a20819d1b155b"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "60f8f9504c17b6186f6d8cbf64ada02c"
server_key_iv: "c317b0fe3c17e3e803b1732c52fcdbeb"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.031570 27875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "8c9a9dc6a89f404cb22a20819d1b155b"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "60f8f9504c17b6186f6d8cbf64ada02c"
server_key_iv: "c317b0fe3c17e3e803b1732c52fcdbeb"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.032859 27875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:48.033563 27890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.033762 27875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:48.033860 27875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "8c9a9dc6a89f404cb22a20819d1b155b"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "60f8f9504c17b6186f6d8cbf64ada02c"
server_key_iv: "c317b0fe3c17e3e803b1732c52fcdbeb"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.033951 27875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:48.050654 27875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:48.053022 27875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:48.053195 27875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:48.057022 27875 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:41213
I20260923 15:11:48.057065 27942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:41213 every 8 connection(s)
I20260923 15:11:48.057452 27875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:48.058254 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27875
I20260923 15:11:48.058377 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:48.058434 27943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:48.058609 22321 external_mini_cluster.cc:1428] Setting key 4ad2d37a663d9c324547a6954e878a06
I20260923 15:11:48.061846 27943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b: Bootstrap starting.
I20260923 15:11:48.062546 27943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:48.062840 27943 log.cc:824] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:48.063529 27943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b: No bootstrap required, opened a new log
I20260923 15:11:48.065392 27943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41213 } }
I20260923 15:11:48.065567 27943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:48.065609 27943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9a9dc6a89f404cb22a20819d1b155b, State: Initialized, Role: FOLLOWER
I20260923 15:11:48.065743 27943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41213 } }
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:48.065822 27943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:48.065860 27943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:48.065943 27943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:48.066684 27943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41213 } }
I20260923 15:11:48.066802 27943 leader_election.cc:302] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c9a9dc6a89f404cb22a20819d1b155b; no voters:
I20260923 15:11:48.066963 27943 leader_election.cc:288] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:48.067026 27948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:48.067180 27948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [term 1 LEADER]: Becoming Leader. State: Replica: 8c9a9dc6a89f404cb22a20819d1b155b, State: Running, Role: LEADER
I20260923 15:11:48.067236 27943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:48.067267 27948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41213 } }
I20260923 15:11:48.067589 27949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41213 } } }
I20260923 15:11:48.067610 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9a9dc6a89f404cb22a20819d1b155b. Latest consensus state: current_term: 1 leader_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9a9dc6a89f404cb22a20819d1b155b" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41213 } } }
I20260923 15:11:48.067729 27949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:48.067742 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:48.067984 27956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:48.068509 27956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:48.070235 27956 catalog_manager.cc:1347] Generated new cluster ID: 0b549983075d48f99a4c8b70358afe04
I20260923 15:11:48.070279 27956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:48.074070 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.059149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.21.204.126:41213)
0923 15:11:48.059349 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.059359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.059392 (+ 33us) server_negotiation.cc:407] Connection header received
0923 15:11:48.060092 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.060102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.060167 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.060331 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.060950 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.062341 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.063246 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.063727 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.063854 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.066376 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:48.066380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:48.066380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:48.066390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:48.068591 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.069020 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.069023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.069024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.069127 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.069374 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.069375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.069375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.069655 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:48.069765 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:48.070049 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 15:11:48.070185 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260923 15:11:48.081719 27956 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:48.082486 27956 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:48.091699 27956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c9a9dc6a89f404cb22a20819d1b155b: Generated new TSK 0
I20260923 15:11:48.091940 27956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:11:48.120481 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41213
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:48.202533 27969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:48.202767 27969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:48.202802 27969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:48.203955 27969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:48.204023 27969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:48.204069 27969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:48.205581 27969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:48.205853 27969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:48.206121 27969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:48.211143 27975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:48.211210 27977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:48.211139 27974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:48.211380 27969 server_base.cc:1034] running on GCE node
I20260923 15:11:48.211586 27969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:48.211819 27969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:48.212980 27969 hybrid_clock.cc:648] HybridClock initialized: now 1790176308212964 us; error 25 us; skew 500 ppm
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.215401 27969 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:11:48.215876 27969 webserver.cc:460] Webserver started at http://127.21.204.65:36267/ using document root <none> and password file <none>
I20260923 15:11:48.216111 27969 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:48.216179 27969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:48.216334 27969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:48.217324 27969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "69047d6bf07d4f1983a37a8a14d3fa7f"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "74af5bce05d6c4cafad32d2d56f6095f"
server_key_iv: "294d8b2286c9bb709539d2cf737a0b02"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.217671 27969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "69047d6bf07d4f1983a37a8a14d3fa7f"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "74af5bce05d6c4cafad32d2d56f6095f"
server_key_iv: "294d8b2286c9bb709539d2cf737a0b02"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.219031 27969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:48.219801 27984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.220001 27969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:48.220100 27969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "69047d6bf07d4f1983a37a8a14d3fa7f"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "74af5bce05d6c4cafad32d2d56f6095f"
server_key_iv: "294d8b2286c9bb709539d2cf737a0b02"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.220186 27969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:48.231575 27969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:48.233448 27969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:48.233606 27969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:48.233834 27969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:48.234165 27969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:48.234220 27969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.234256 27969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:48.234287 27969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.240381 27969 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:44399
I20260923 15:11:48.240455 28097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:44399 every 8 connection(s)
I20260923 15:11:48.240778 27969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:48.245918 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 27969
I20260923 15:11:48.246038 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:48.246254 22321 external_mini_cluster.cc:1428] Setting key 5e8571e42ffceee0d0f907077cdc2375
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:48.251514 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.241539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:40083 (local address 127.21.204.126:41213)
0923 15:11:48.241605 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.241607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.241823 (+ 216us) server_negotiation.cc:407] Connection header received
0923 15:11:48.242815 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.242816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.242830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.242893 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.243515 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.244022 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.245047 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.245267 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.245345 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.247858 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:48.247860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:48.247860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:48.247869 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:48.249603 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.250308 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.250310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.250310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.250367 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.250729 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.250730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.250730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.250845 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:48.250981 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:48.251308 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 15:11:48.251400 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:11:48.251717 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.241529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41213 (local address 127.21.204.65:40083)
0923 15:11:48.241704 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:48.241740 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:48.242700 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:48.242964 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:48.242968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:48.243225 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:48.243413 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:48.243417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.244126 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.244127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:48.244942 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:48.244945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.245354 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.245356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:48.245485 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.246160 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:48.246169 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:48.247755 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:48.249709 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:48.249717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:48.249719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:48.250083 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:48.250445 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:48.250446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:48.250447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:48.250598 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:48.250972 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:48.250980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:48.251159 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 15:11:48.251269 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 15:11:48.251434 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260923 15:11:48.252282 28098 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41213
I20260923 15:11:48.252410 28098 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:48.252624 28098 heartbeater.cc:511] Master 127.21.204.126:41213 requested a full tablet report, sending...
I20260923 15:11:48.253181 27907 ts_manager.cc:194] Registered new tserver with Master: 69047d6bf07d4f1983a37a8a14d3fa7f (127.21.204.65:44399)
I20260923 15:11:48.253782 27907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:40083
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:11:48.273118 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41213
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:48.353660 28103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:48.353894 28103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:48.353952 28103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:48.355125 28103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:48.355192 28103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:48.355239 28103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:48.356736 28103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:48.356999 28103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:48.357234 28103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:48.362341 28108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:48.362347 28111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:48.362350 28109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:48.362633 28103 server_base.cc:1034] running on GCE node
I20260923 15:11:48.362864 28103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:48.363123 28103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:48.364292 28103 hybrid_clock.cc:648] HybridClock initialized: now 1790176308364283 us; error 41 us; skew 500 ppm
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.366833 28103 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:11:48.367403 28103 webserver.cc:460] Webserver started at http://127.21.204.66:40097/ using document root <none> and password file <none>
I20260923 15:11:48.367663 28103 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:48.367739 28103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:48.367877 28103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:48.368997 28103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "9909782788634394a43cdef11e94f43b"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "36c52cf7db473f1edcbd2d6b693321c2"
server_key_iv: "10f7d5165971f61c5eceeb45cdc983c3"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.369374 28103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "9909782788634394a43cdef11e94f43b"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "36c52cf7db473f1edcbd2d6b693321c2"
server_key_iv: "10f7d5165971f61c5eceeb45cdc983c3"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.370783 28103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:48.371556 28118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.371760 28103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:48.371866 28103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "9909782788634394a43cdef11e94f43b"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "36c52cf7db473f1edcbd2d6b693321c2"
server_key_iv: "10f7d5165971f61c5eceeb45cdc983c3"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.371939 28103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:48.389643 28103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:48.391744 28103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:48.391916 28103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:48.392166 28103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:48.392486 28103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:48.392541 28103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.392582 28103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:48.392612 28103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.399425 28103 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:35391
I20260923 15:11:48.399466 28231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:35391 every 8 connection(s)
I20260923 15:11:48.399811 28103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:48.407815 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 28103
I20260923 15:11:48.407946 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:48.408140 22321 external_mini_cluster.cc:1428] Setting key 1cef06ddf16d1534f697074143190be8
I20260923 15:11:48.408916 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.400568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:38663 (local address 127.21.204.126:41213)
0923 15:11:48.400627 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.400629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.400804 (+ 175us) server_negotiation.cc:407] Connection header received
0923 15:11:48.401723 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.401725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.401736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.401793 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.402440 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.402967 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.403910 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.404098 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.404171 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.406128 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:48.406129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:48.406129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:48.406136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:48.407360 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.407796 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.407797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.407797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.407849 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.408183 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.408185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.408185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.408328 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:48.408411 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:48.408718 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 15:11:48.408823 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 15:11:48.409086 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.400518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41213 (local address 127.21.204.66:38663)
0923 15:11:48.400673 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:48.400714 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:48.401610 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:48.401853 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:48.401856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:48.402162 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:48.402359 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:48.402364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.403047 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.403048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:48.403826 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:48.403828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.404180 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.404180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:48.404287 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.404704 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:48.404712 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:48.406048 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:48.407434 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:48.407440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:48.407442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:48.407719 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:48.407944 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:48.407945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:48.407945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:48.408105 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:48.408429 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:48.408436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:48.408580 (+ 144us) client_negotiation.cc:769] Sending connection context
0923 15:11:48.408670 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 15:11:48.408853 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260923 15:11:48.409552 28232 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41213
I20260923 15:11:48.409659 28232 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:48.409832 28232 heartbeater.cc:511] Master 127.21.204.126:41213 requested a full tablet report, sending...
I20260923 15:11:48.410257 27907 ts_manager.cc:194] Registered new tserver with Master: 9909782788634394a43cdef11e94f43b (127.21.204.66:35391)
I20260923 15:11:48.410797 27907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:38663
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:11:48.432915 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41213
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:48.513633 28237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:48.513832 28237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:48.513855 28237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:48.515002 28237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:48.515043 28237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:48.515069 28237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:48.516496 28237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:48.516670 28237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:48.516877 28237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:48.521953 28242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:48.521994 28245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:48.521960 28243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:48.522265 28237 server_base.cc:1034] running on GCE node
I20260923 15:11:48.522450 28237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:48.522652 28237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:48.523814 28237 hybrid_clock.cc:648] HybridClock initialized: now 1790176308523799 us; error 29 us; skew 500 ppm
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.526510 28237 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:11:48.527024 28237 webserver.cc:460] Webserver started at http://127.21.204.67:38513/ using document root <none> and password file <none>
I20260923 15:11:48.527300 28237 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:48.527349 28237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:48.527529 28237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:48.528661 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "eb753ba43c2f4c74b5c5763c1b5eed08"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "0063126c142229a54752805ca2411234"
server_key_iv: "9ce74ab652f5c71d31063100719489ec"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.529032 28237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "eb753ba43c2f4c74b5c5763c1b5eed08"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "0063126c142229a54752805ca2411234"
server_key_iv: "9ce74ab652f5c71d31063100719489ec"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.530638 28237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:48.531615 28252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.531841 28237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:48.531939 28237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "eb753ba43c2f4c74b5c5763c1b5eed08"
format_stamp: "Formatted at 2026-09-23 15:11:48 on dist-test-slave-t5lt"
server_key: "0063126c142229a54752805ca2411234"
server_key_iv: "9ce74ab652f5c71d31063100719489ec"
server_key_version: "encryptionkey@0"
I20260923 15:11:48.532016 28237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:48.548677 28237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:48.550997 28237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:48.551156 28237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:48.551409 28237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:48.551736 28237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:48.551788 28237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.551831 28237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:48.551859 28237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:48.557921 28237 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:37327
I20260923 15:11:48.557951 28365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:37327 every 8 connection(s)
I20260923 15:11:48.558271 28237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:11:48.567992 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 28237
I20260923 15:11:48.568104 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:48.568120 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.558984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:37329 (local address 127.21.204.126:41213)
0923 15:11:48.559071 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.559073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.559264 (+ 191us) server_negotiation.cc:407] Connection header received
0923 15:11:48.560222 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.560223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.560236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.560285 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.560898 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.561410 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.562472 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.562722 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.562811 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.564887 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:48.564888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:48.564889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:48.564896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:48.566348 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.566887 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.566888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.566888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.566961 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.567263 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.567264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.567265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.567397 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:48.567499 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:48.567852 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 15:11:48.567964 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 15:11:48.568243 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.558946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41213 (local address 127.21.204.67:37329)
0923 15:11:48.559123 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:48.559180 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:48.560100 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:48.560371 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:48.560375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:48.560622 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:48.560807 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:48.560811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.561503 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.561504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:48.562380 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:48.562383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.562824 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.562825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:48.562948 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.563449 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:48.563458 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:48.564787 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:48.566457 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:48.566465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:48.566467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:48.566780 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:48.567053 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:48.567054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:48.567054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:48.567184 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:48.567489 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:48.567497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:48.567646 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 15:11:48.567777 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 15:11:48.567960 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260923 15:11:48.568329 22321 external_mini_cluster.cc:1428] Setting key 2a4938463e08038f6d78aa76886b381e
I20260923 15:11:48.568845 28366 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41213
I20260923 15:11:48.568975 28366 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:48.569185 28366 heartbeater.cc:511] Master 127.21.204.126:41213 requested a full tablet report, sending...
I20260923 15:11:48.569727 27907 ts_manager.cc:194] Registered new tserver with Master: eb753ba43c2f4c74b5c5763c1b5eed08 (127.21.204.67:37327)
I20260923 15:11:48.570150 27907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:37329
I20260923 15:11:48.580667 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/test-admin.keytab.
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.598918 22321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 15:11:48.600646 28373 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:48.599839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.21.204.65:44399)
0923 15:11:48.600024 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.600029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.600038 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:48.600063 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.600064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.600078 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.600164 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.600554 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55284: BlockingRecv error: recv got EOF from 127.0.0.1:55284 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
W20260923 15:11:48.600736 22321 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5cc' of type FILE
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.601727 22321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.602671 22321 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:48.611075 28373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.603206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.21.204.65:44399)
0923 15:11:48.603267 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.603269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.603343 (+ 74us) server_negotiation.cc:407] Connection header received
0923 15:11:48.603395 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.603395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.603405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.603458 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.603920 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.604633 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.605399 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.605705 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.605760 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.608165 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:48.608175 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:48.608176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:48.608188 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:48.609816 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.610243 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.610251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.610251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.610306 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.610508 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.610509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.610509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.610654 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:48.610800 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:48.610883 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 15:11:48.610962 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 15:11:48.613013 28373 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:48.612377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.21.204.65:44399)
0923 15:11:48.612430 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.612432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.612540 (+ 108us) server_negotiation.cc:407] Connection header received
0923 15:11:48.612621 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.612622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.612634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.612700 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.612968 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55300: BlockingRecv error: recv got EOF from 127.0.0.1:55300 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260923 15:11:48.613145 22321 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5cc' of type FILE
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.614154 22321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.614969 22321 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:48.622041 28373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.615452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.21.204.65:44399)
0923 15:11:48.615508 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.615509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.615663 (+ 154us) server_negotiation.cc:407] Connection header received
0923 15:11:48.615696 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.615697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.615706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.615755 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.616087 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.616607 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.617330 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.617519 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.617572 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.619681 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:48.619682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:48.619682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:48.619689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:48.620905 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.621221 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.621222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.621222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.621280 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.621520 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.621521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.621522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.621635 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:48.621715 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:48.621810 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 15:11:48.621869 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260923 15:11:48.623690 28373 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:48.623099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.21.204.65:44399)
0923 15:11:48.623170 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.623173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.623182 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:48.623257 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.623258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.623265 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.623313 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.623630 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55308: BlockingRecv error: recv got EOF from 127.0.0.1:55308 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 15:11:48.623723 22321 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790176278407460-22321-0/krb5kdc/krb5cc' of type FILE
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.624680 22321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:48.625623 22321 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[27862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:11:48.632468 28373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:48.626218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.21.204.65:44399)
0923 15:11:48.626272 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:48.626274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:48.626299 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:11:48.626371 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:48.626372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:48.626381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:48.626445 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:48.626812 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.627266 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.627955 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:48.628128 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:48.628215 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:48.630219 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:48.630220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:48.630220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:48.630228 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:48.631412 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.631752 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.631753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.631754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.631810 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:48.632008 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:48.632009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:48.632009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:48.632104 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:48.632211 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:48.632308 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 15:11:48.632378 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:11:48.633108 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27969
I20260923 15:11:48.638573 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 28103
I20260923 15:11:48.643050 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 28237
I20260923 15:11:48.647449 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 27875
2026-09-23T15:11:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1883 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[28386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[28386](info): set up 2 sockets
Sep 23 15:11:48 dist-test-slave-t5lt krb5kdc[28386](info): commencing operation
krb5kdc: starting...
W20260923 15:11:49.751569 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:49 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab.
Entry for principal oryx/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab.
I20260923 15:11:49.877156 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40489
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab
--principal=oryx/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:40489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:49.962074 28399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:49.962316 28399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:49.962352 28399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:49.963497 28399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:49.963562 28399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:49.963588 28399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:49.963618 28399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:49.963641 28399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:49.965102 28399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:40489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab
--principal=oryx/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:49.965381 28399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:49.965605 28399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:11:49.970980 28399 server_base.cc:1034] running on GCE node
W20260923 15:11:49.970945 28405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:49.970945 28404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:49.970938 28407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:49.971421 28399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:49.971866 28399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:49.973083 28399 hybrid_clock.cc:648] HybridClock initialized: now 1790176309973052 us; error 62 us; skew 500 ppm
Sep 23 15:11:49 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176309, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:49.975786 28399 init.cc:377] Logged in from keytab as oryx/127.21.204.126@KRBTEST.COM (short username oryx)
I20260923 15:11:49.976307 28399 webserver.cc:460] Webserver started at http://127.21.204.126:39395/ using document root <none> and password file <none>
I20260923 15:11:49.976547 28399 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:49.976617 28399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:49.976776 28399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:49.977923 28399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "03997f227c114b958667f054eaa101f6"
format_stamp: "Formatted at 2026-09-23 15:11:49 on dist-test-slave-t5lt"
server_key: "a6b9b56949035d92f36d2f39bbcbb5f7"
server_key_iv: "ea64950367f8d381f7cb7d551527ee2c"
server_key_version: "encryptionkey@0"
I20260923 15:11:49.978323 28399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "03997f227c114b958667f054eaa101f6"
format_stamp: "Formatted at 2026-09-23 15:11:49 on dist-test-slave-t5lt"
server_key: "a6b9b56949035d92f36d2f39bbcbb5f7"
server_key_iv: "ea64950367f8d381f7cb7d551527ee2c"
server_key_version: "encryptionkey@0"
I20260923 15:11:49.979759 28399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 15:11:49.980657 28414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:49.980872 28399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:49.980984 28399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "03997f227c114b958667f054eaa101f6"
format_stamp: "Formatted at 2026-09-23 15:11:49 on dist-test-slave-t5lt"
server_key: "a6b9b56949035d92f36d2f39bbcbb5f7"
server_key_iv: "ea64950367f8d381f7cb7d551527ee2c"
server_key_version: "encryptionkey@0"
I20260923 15:11:49.981060 28399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:49.997248 28399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:49.999615 28399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:49.999800 28399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:50.004284 28399 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:40489
I20260923 15:11:50.004320 28466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:40489 every 8 connection(s)
I20260923 15:11:50.004701 28399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:50.005750 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:50.008104 28467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: Bootstrap starting.
I20260923 15:11:50.008661 28467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:50.008893 28467 log.cc:824] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:50.009668 28467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: No bootstrap required, opened a new log
I20260923 15:11:50.011397 28467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } }
I20260923 15:11:50.011642 28467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:50.011703 28467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03997f227c114b958667f054eaa101f6, State: Initialized, Role: FOLLOWER
I20260923 15:11:50.011834 28467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } }
I20260923 15:11:50.011904 28467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:50.011925 28467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:50.011989 28467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:50.012180 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 28399
I20260923 15:11:50.012303 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:50.012506 22321 external_mini_cluster.cc:1428] Setting key 8c939f43632977b8d947051391e19fdd
I20260923 15:11:50.013283 28467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } }
I20260923 15:11:50.013386 28467 leader_election.cc:302] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03997f227c114b958667f054eaa101f6; no voters:
I20260923 15:11:50.013571 28467 leader_election.cc:288] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:50.013621 28472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:50.013762 28472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 1 LEADER]: Becoming Leader. State: Replica: 03997f227c114b958667f054eaa101f6, State: Running, Role: LEADER
I20260923 15:11:50.013864 28472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } }
I20260923 15:11:50.013856 28467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:50.014324 28474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03997f227c114b958667f054eaa101f6. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } }
I20260923 15:11:50.014412 28474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:50.014345 28473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } }
I20260923 15:11:50.014562 28473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:50.015348 28476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:50.015870 28476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:50.017632 28476 catalog_manager.cc:1347] Generated new cluster ID: d1f82c282da24552925d540452d15f68
I20260923 15:11:50.017692 28476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:50.023957 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.013056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.21.204.126:40489)
0923 15:11:50.013245 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.013255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.013291 (+ 36us) server_negotiation.cc:407] Connection header received
0923 15:11:50.013420 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.013427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.013494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.013665 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.014608 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.016133 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.017207 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.017771 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.018248 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.020623 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.020627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.020627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.020638 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.022437 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.022810 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.022819 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.022819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.022912 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.023124 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.023125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.023125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.023365 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.023549 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.023615 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.023761 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260923 15:11:50.029044 28476 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:50.029592 28476 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:50.031536 28476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: Generated new TSK 0
I20260923 15:11:50.031716 28476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab.
Entry for principal oryx/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab.
I20260923 15:11:50.069034 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40489
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab
--principal=oryx/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:50.156277 28493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:50.156483 28493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:50.156509 28493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:50.157737 28493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:50.157779 28493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:50.157806 28493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:50.159264 28493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab
--principal=oryx/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:50.159528 28493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:50.159781 28493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:50.165102 28501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.165140 28499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.165303 28493 server_base.cc:1034] running on GCE node
W20260923 15:11:50.165151 28498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.165648 28493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:50.165961 28493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:50.167124 28493 hybrid_clock.cc:648] HybridClock initialized: now 1790176310167111 us; error 30 us; skew 500 ppm
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:50.174340 28493 init.cc:377] Logged in from keytab as oryx/127.21.204.65@KRBTEST.COM (short username oryx)
I20260923 15:11:50.174757 28493 webserver.cc:460] Webserver started at http://127.21.204.65:36207/ using document root <none> and password file <none>
I20260923 15:11:50.175006 28493 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:50.175051 28493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:50.175222 28493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:50.176322 28493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "4ce281b9d1e4434cb2a07ed3f186cb24"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "2d5707aadfb69c74b14966469d4b6f73"
server_key_iv: "1478528aca4453081d63a43b7ea5c274"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.176704 28493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "4ce281b9d1e4434cb2a07ed3f186cb24"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "2d5707aadfb69c74b14966469d4b6f73"
server_key_iv: "1478528aca4453081d63a43b7ea5c274"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.178265 28493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260923 15:11:50.179085 28508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.179309 28493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:50.179399 28493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "4ce281b9d1e4434cb2a07ed3f186cb24"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "2d5707aadfb69c74b14966469d4b6f73"
server_key_iv: "1478528aca4453081d63a43b7ea5c274"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.179454 28493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:50.195433 28493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:50.197688 28493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:50.197799 28493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:50.198064 28493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:50.198424 28493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:50.198460 28493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.198483 28493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:50.198531 28493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.204250 28493 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:44703
I20260923 15:11:50.204350 28621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:44703 every 8 connection(s)
I20260923 15:11:50.204651 28493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:50.214279 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.205437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:47775 (local address 127.21.204.126:40489)
0923 15:11:50.205506 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.205509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.205768 (+ 259us) server_negotiation.cc:407] Connection header received
0923 15:11:50.206272 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.206273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.206283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.206335 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.207220 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.207763 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.208783 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.208991 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.209056 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.211310 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.211311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.211311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.211319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.212942 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.213338 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.213339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.213339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.213389 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.213686 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.213687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.213688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.213806 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.213862 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.214145 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.214215 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:11:50.214463 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.205414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:47775)
0923 15:11:50.205648 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.205684 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.206188 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.206387 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.206390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.206943 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.207140 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.207144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.207879 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.207880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.208671 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.208673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.209097 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.209098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.209201 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.209721 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.209730 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.211213 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.213028 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.213035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.213038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.213268 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.213452 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.213453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.213453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.213587 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.213871 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.213901 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.214016 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.214115 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.214239 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260923 15:11:50.214560 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 28493
I20260923 15:11:50.214658 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:50.214891 22321 external_mini_cluster.cc:1428] Setting key 077d2d80f59cb65e9b634c6cb7614559
I20260923 15:11:50.215353 28622 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40489
I20260923 15:11:50.215449 28622 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:50.215716 28622 heartbeater.cc:511] Master 127.21.204.126:40489 requested a full tablet report, sending...
I20260923 15:11:50.216269 28431 ts_manager.cc:194] Registered new tserver with Master: 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703)
I20260923 15:11:50.216815 28431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:47775
WARNING: no policy specified for oryx/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab.
Entry for principal oryx/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab.
I20260923 15:11:50.242141 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40489
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab
--principal=oryx/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:50.324733 28627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:50.324949 28627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:50.324971 28627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:50.326177 28627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:50.326217 28627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:50.326243 28627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:50.327714 28627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab
--principal=oryx/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:50.327975 28627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:50.328198 28627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:50.333388 28635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.333419 28632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.333501 28627 server_base.cc:1034] running on GCE node
W20260923 15:11:50.333583 28633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.333915 28627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:50.334182 28627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:50.335345 28627 hybrid_clock.cc:648] HybridClock initialized: now 1790176310335331 us; error 45 us; skew 500 ppm
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:50.337719 28627 init.cc:377] Logged in from keytab as oryx/127.21.204.66@KRBTEST.COM (short username oryx)
I20260923 15:11:50.338209 28627 webserver.cc:460] Webserver started at http://127.21.204.66:43155/ using document root <none> and password file <none>
I20260923 15:11:50.338456 28627 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:50.338533 28627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:50.338685 28627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:50.339689 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "9b6dc5f13a1341dc9fc047f5f112bb74"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "5e0c235223406b6e89a2f06f84281b8f"
server_key_iv: "a0e43d1df065b9d707574bac841b6564"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.340047 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "9b6dc5f13a1341dc9fc047f5f112bb74"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "5e0c235223406b6e89a2f06f84281b8f"
server_key_iv: "a0e43d1df065b9d707574bac841b6564"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.341362 28627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:50.342188 28642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.342355 28627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:11:50.342463 28627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "9b6dc5f13a1341dc9fc047f5f112bb74"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "5e0c235223406b6e89a2f06f84281b8f"
server_key_iv: "a0e43d1df065b9d707574bac841b6564"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.342538 28627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:50.349593 28627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:50.351666 28627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:50.351840 28627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:50.352072 28627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:50.352383 28627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:50.352438 28627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.352479 28627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:50.352506 28627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.358496 28627 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:42863
I20260923 15:11:50.358549 28755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:42863 every 8 connection(s)
I20260923 15:11:50.358882 28627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:50.367100 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 28627
I20260923 15:11:50.367213 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:50.367422 22321 external_mini_cluster.cc:1428] Setting key 74260978096a4144a388da45ae0231a5
I20260923 15:11:50.368263 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.359572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:54713 (local address 127.21.204.126:40489)
0923 15:11:50.359620 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.359622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.359811 (+ 189us) server_negotiation.cc:407] Connection header received
0923 15:11:50.360266 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.360277 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.360287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.360344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.361230 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.362120 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.363135 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.363323 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.363377 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.365316 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.365317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.365317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.365324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.366763 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.367106 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.367107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.367108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.367169 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.367592 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.367593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.367594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.367725 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.367809 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.368084 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.368169 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:11:50.368474 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.359529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:54713)
0923 15:11:50.359684 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.359725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.360188 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.360393 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.360396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.360933 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.361137 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.361141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.362217 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.362218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.363055 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.363058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.363398 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.363399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.363502 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.363915 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.363923 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.365223 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.366834 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.366841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.366843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.367046 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.367264 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.367265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.367266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.367424 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.367786 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.367793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.367929 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.368039 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.368220 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260923 15:11:50.369825 28756 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40489
I20260923 15:11:50.369943 28756 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:50.370124 28756 heartbeater.cc:511] Master 127.21.204.126:40489 requested a full tablet report, sending...
I20260923 15:11:50.370532 28431 ts_manager.cc:194] Registered new tserver with Master: 9b6dc5f13a1341dc9fc047f5f112bb74 (127.21.204.66:42863)
I20260923 15:11:50.371043 28431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:54713
WARNING: no policy specified for oryx/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab.
Entry for principal oryx/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab.
I20260923 15:11:50.394040 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40489
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab
--principal=oryx/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:50.476727 28761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:50.476964 28761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:50.477005 28761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:50.478248 28761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:50.478307 28761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:50.478353 28761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:50.479815 28761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab
--principal=oryx/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:50.480080 28761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:50.480306 28761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:50.485512 28766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.485553 28767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.485553 28769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.485975 28761 server_base.cc:1034] running on GCE node
I20260923 15:11:50.486212 28761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:50.486477 28761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:50.487674 28761 hybrid_clock.cc:648] HybridClock initialized: now 1790176310487647 us; error 45 us; skew 500 ppm
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:50.490444 28761 init.cc:377] Logged in from keytab as oryx/127.21.204.67@KRBTEST.COM (short username oryx)
I20260923 15:11:50.490823 28761 webserver.cc:460] Webserver started at http://127.21.204.67:37691/ using document root <none> and password file <none>
I20260923 15:11:50.491027 28761 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:50.491070 28761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:50.491163 28761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:50.492115 28761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "9d14836f0e77457ca7a893e002d211da"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "c13b0db34f2cc258988d7f5b55c9ca04"
server_key_iv: "359dec5571a943861b5f44d6fabac3c2"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.492412 28761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "9d14836f0e77457ca7a893e002d211da"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "c13b0db34f2cc258988d7f5b55c9ca04"
server_key_iv: "359dec5571a943861b5f44d6fabac3c2"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.493618 28761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:11:50.494342 28776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.494504 28761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.494588 28761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "9d14836f0e77457ca7a893e002d211da"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "c13b0db34f2cc258988d7f5b55c9ca04"
server_key_iv: "359dec5571a943861b5f44d6fabac3c2"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.494658 28761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:50.507282 28761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:50.509284 28761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:50.509445 28761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:50.509685 28761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:50.510043 28761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:50.510097 28761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.510138 28761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:50.510165 28761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.516016 28761 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:34017
I20260923 15:11:50.516053 28889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:34017 every 8 connection(s)
I20260923 15:11:50.516438 28761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:50.518628 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 28761
I20260923 15:11:50.518738 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:50.518929 22321 external_mini_cluster.cc:1428] Setting key eb1127996506e872b2a755717fe3e02e
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:50.525859 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.517145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:60717 (local address 127.21.204.126:40489)
0923 15:11:50.517186 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.517188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.517403 (+ 215us) server_negotiation.cc:407] Connection header received
0923 15:11:50.517972 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.517974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.517987 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.518050 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.519248 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.519773 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.520749 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.520960 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.521044 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.523089 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.523090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.523091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.523098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.524519 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.524975 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.524976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.524977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.525030 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.525265 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.525266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.525266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.525400 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.525451 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.525732 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.525794 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:11:50.526120 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.517121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:60717)
0923 15:11:50.517291 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.517325 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.517826 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.518117 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.518120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.518857 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.519143 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.519148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.519868 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.519869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.520642 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.520644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.521054 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.521055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.521186 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.521613 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.521620 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.523009 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.524677 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.524683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.524685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.524900 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.525108 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.525108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.525109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.525207 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.525458 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.525463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.525578 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.525699 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.525816 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260923 15:11:50.526643 28890 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40489
I20260923 15:11:50.526723 28890 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:50.526916 28890 heartbeater.cc:511] Master 127.21.204.126:40489 requested a full tablet report, sending...
I20260923 15:11:50.527319 28431 ts_manager.cc:194] Registered new tserver with Master: 9d14836f0e77457ca7a893e002d211da (127.21.204.67:34017)
I20260923 15:11:50.527714 28431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:60717
I20260923 15:11:50.530529 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.125.keytab.
Entry for principal oryx/127.21.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.125.keytab.
I20260923 15:11:50.557773 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:39959
--webserver_interface=127.21.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.125.keytab
--principal=oryx/127.21.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:40489,127.21.204.125:39959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:50.638239 28895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:50.638446 28895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:50.638469 28895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:50.639621 28895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:50.639664 28895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:50.639679 28895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:50.639693 28895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:50.639746 28895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:50.641175 28895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:40489,127.21.204.125:39959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.125.keytab
--principal=oryx/127.21.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
W20260923 15:11:50.641265 28895 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.204.126:40489,127.21.204.125:39959'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 15:11:50.641433 28895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:50.641680 28895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:50.646715 28901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.646708 28903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.646893 28895 server_base.cc:1034] running on GCE node
W20260923 15:11:50.646709 28900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.647163 28895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:50.647464 28895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:50.648653 28895 hybrid_clock.cc:648] HybridClock initialized: now 1790176310648599 us; error 47 us; skew 500 ppm
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:50.651153 28895 init.cc:377] Logged in from keytab as oryx/127.21.204.125@KRBTEST.COM (short username oryx)
I20260923 15:11:50.651609 28895 webserver.cc:460] Webserver started at http://127.21.204.125:33883/ using document root <none> and password file <none>
I20260923 15:11:50.651855 28895 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:50.651898 28895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:50.652050 28895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:50.653090 28895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/instance:
uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "e5c7ca50c5d61c0b5b7d4539b551828e"
server_key_iv: "ae710c60b774f780b2d9049268156ea3"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.653436 28895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal/instance:
uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "e5c7ca50c5d61c0b5b7d4539b551828e"
server_key_iv: "ae710c60b774f780b2d9049268156ea3"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.655212 28895 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260923 15:11:50.655906 28910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.656088 28895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:50.656176 28895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "e5c7ca50c5d61c0b5b7d4539b551828e"
server_key_iv: "ae710c60b774f780b2d9049268156ea3"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.656244 28895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:50.665102 28895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:50.667330 28895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:50.667474 28895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:50.671936 28895 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.125:39959
I20260923 15:11:50.671994 28962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.125:39959 every 8 connection(s)
I20260923 15:11:50.672365 28895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
I20260923 15:11:50.673391 28963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:50.674991 28963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.125@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:50.682562 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 28895
I20260923 15:11:50.684604 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.675820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.21.204.126:40489)
0923 15:11:50.675892 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.675895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.676058 (+ 163us) server_negotiation.cc:407] Connection header received
0923 15:11:50.676693 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.676695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.676706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.676771 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.677389 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.677970 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.679065 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.679246 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.679300 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.681411 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.681413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.681413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.681419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.682992 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.683540 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.683541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.683542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.683606 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.683871 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.683873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.683873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.684044 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.684128 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.684466 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.684528 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:11:50.684825 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.675728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:39736)
0923 15:11:50.675930 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.675971 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.676415 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.676847 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.676851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.677090 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.677290 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.677294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.678062 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.678063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.678917 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.678920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.679320 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.679321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.679425 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.679958 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.679967 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.681329 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.683085 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.683091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.683096 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.683449 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.683675 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.683676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.683676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.683810 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.684118 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.684124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.684306 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.684439 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.684599 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260923 15:11:50.685436 28963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.204.125@KRBTEST.COM
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.125@KRBTEST.COM for oryx/127.21.204.125@KRBTEST.COM
I20260923 15:11:50.693086 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.685731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.0.0.1:33916)
0923 15:11:50.685779 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.685788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.685825 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.686070 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.686070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.686223 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.686264 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.686266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.687111 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.687112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.687584 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.687585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.687974 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.687975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.688042 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.688457 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.688464 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.689913 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.691968 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.691969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.691969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.692154 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.692370 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.692371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.692371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.692427 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.692738 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.692739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.692799 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.692867 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.692963 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 15:11:50.693243 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.683039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.21.204.125:39959)
0923 15:11:50.683216 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.683220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.683387 (+ 167us) server_negotiation.cc:407] Connection header received
0923 15:11:50.683415 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.683418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.683433 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.683518 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.684366 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.685461 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.686385 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.686848 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.686917 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.689567 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.689575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.689575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.689582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.691915 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.692312 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.692313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.692313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.692388 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.692609 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.692610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.692610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.692722 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.692867 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.692968 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.693097 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260923 15:11:50.693431 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.685805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.21.204.125:39959)
0923 15:11:50.685956 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.685958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.685967 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:50.685986 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.685987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.685994 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.686069 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.686344 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.687030 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.687685 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.687904 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.687976 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.690004 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.690005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.690005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.690012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.691887 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.692230 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.692233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.692234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.692304 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.692492 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.692494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.692494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.692664 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.692741 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.692916 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.693037 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260923 15:11:50.694764 28963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Bootstrap starting.
I20260923 15:11:50.695326 28963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:50.695542 28963 log.cc:824] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:50.696185 28963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: No bootstrap required, opened a new log
I20260923 15:11:50.697657 28963 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } }
I20260923 15:11:50.697779 28963 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:50.697803 28963 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb48151fe57d4268b99e218f1bcfd2ba, State: Initialized, Role: FOLLOWER
I20260923 15:11:50.697940 28963 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } }
I20260923 15:11:50.698226 28963 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:50.698218 28970 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } } }
I20260923 15:11:50.698347 28970 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: FOLLOWER
W20260923 15:11:50.699890 28981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:11:50.699944 28981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:11:50.701443 28895 master_runner.cc:428] Detected that this master eb48151fe57d4268b99e218f1bcfd2ba is joining an existing cluster
I20260923 15:11:50.701491 28895 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.204.125:39959
I20260923 15:11:50.709241 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.702822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.0.0.1:33930)
0923 15:11:50.702930 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.702938 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.702994 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.703139 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.703140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.703321 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.703377 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.703379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.704262 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.704263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.704828 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.704830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.705303 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.705303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.705355 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.705810 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.705814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.706174 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.708155 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.708157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.708319 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.708511 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.708513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.708598 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.708893 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.708893 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.708949 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.709011 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.709120 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":75,"threads_started":1}
I20260923 15:11:50.709247 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.702822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.21.204.125:39959)
0923 15:11:50.702859 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.702860 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.702976 (+ 116us) server_negotiation.cc:407] Connection header received
0923 15:11:50.703053 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.703054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.703062 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.703088 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.703469 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.704186 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.704932 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.705202 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.705258 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.706267 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.706269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.706269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.706275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.708049 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708396 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.708397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.708398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.708449 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708664 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.708665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.708666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.708796 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.708871 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.709039 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.709167 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":22}
I20260923 15:11:50.709630 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.702591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.21.204.126:40489)
0923 15:11:50.702656 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.702659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.702765 (+ 106us) server_negotiation.cc:407] Connection header received
0923 15:11:50.702876 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.702877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.702890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.702953 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.703263 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.704033 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.705077 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.705377 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.705441 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.706727 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.706729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.706730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.706737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.708375 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708719 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.708720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.708721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.708786 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708991 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.708992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.708992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.709131 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.709197 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.709410 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.709514 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 15:11:50.709949 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.702481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:39738)
0923 15:11:50.702684 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.702696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.702768 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.702955 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.702956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.703127 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.703185 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.703187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.704124 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.704125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.704992 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.704994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.705442 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.705443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.705820 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.706248 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.706253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.706654 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.708458 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.708459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.708659 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.708853 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.708853 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.708854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.708911 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.709226 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.709227 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.709267 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.709712 (+ 445us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.709818 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":122,"threads_started":1}
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176309, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.126@KRBTEST.COM for oryx/127.21.204.125@KRBTEST.COM
I20260923 15:11:50.721352 28968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.712193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.21.204.125:39959)
0923 15:11:50.712226 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.712228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.714088 (+ 1860us) server_negotiation.cc:407] Connection header received
0923 15:11:50.714183 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.714184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.714193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.714244 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.714758 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.715296 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.716430 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.716602 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.716687 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.718865 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.718867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.718867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.718874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.720156 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.720545 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.720546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.720547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.720603 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.720812 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.720813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.720813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.720916 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.720986 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.721184 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.721260 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 15:11:50.721369 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.712111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.0.0.1:33944)
0923 15:11:50.713984 (+ 1873us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.714033 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.714107 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.714248 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.714251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.714585 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.714679 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.714681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.715372 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.715373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.716354 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.716357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.716678 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.716679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.716779 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.717205 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.717211 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.718771 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.720260 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.720268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.720271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.720468 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.720671 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.720672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.720672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.720740 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.720993 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.721000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.721050 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.721145 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.721264 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1842,"thread_start_us":68,"threads_started":1}
I20260923 15:11:50.721984 28431 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.204.125:39959: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
member_type: NON_VOTER
last_known_addr {
host: "127.21.204.125"
port: 39959
}
attrs {
promote: true
}
}
dest_uuid: "03997f227c114b958667f054eaa101f6"
cas_config_opid_index: -1
I20260923 15:11:50.722307 28431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } }
I20260923 15:11:50.722831 28991 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } }
I20260923 15:11:50.723093 28937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:50.723870 28991 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.204.125:39959
I20260923 15:11:50.723960 28937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Refusing update from remote peer 03997f227c114b958667f054eaa101f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 15:11:50.723949 28991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } }
I20260923 15:11:50.724006 28991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:50.724239 28895 master.cc:537] Master@127.21.204.125:39959 shutting down...
I20260923 15:11:50.724349 28990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 15:11:50.725033 28416 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 15:11:50.725464 28416 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 -> Peer eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125:39959): Couldn't send request to peer eb48151fe57d4268b99e218f1bcfd2ba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 15:11:50.726980 28895 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:50.727068 28895 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:50.727120 28895 tablet_replica.cc:330] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: stopping tablet replica
I20260923 15:11:50.740023 28895 master.cc:552] Master@127.21.204.125:39959 shutdown complete.
I20260923 15:11:50.740091 28895 master_runner.cc:305] Clearing existing system tablet
I20260923 15:11:50.740222 28895 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:50.740283 28895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:50.740909 28895 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260923 15:11:50.741420 28995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.741549 28895 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:50.741605 28895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.741636 28895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "e5c7ca50c5d61c0b5b7d4539b551828e"
server_key_iv: "ae710c60b774f780b2d9049268156ea3"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.742333 28895 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:50.743741 28895 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 15:11:50.743829 28895 log.cc:1196] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 15:11:50.744019 28895 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Deleting consensus metadata
I20260923 15:11:50.744304 28895 master_runner.cc:315] Copying system tablet from 127.21.204.126:40489
I20260923 15:11:50.744994 28895 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: tablet copy: Beginning tablet copy session from remote peer at address 127.21.204.126:40489
I20260923 15:11:50.751415 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.745359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.21.204.126:40489)
0923 15:11:50.745406 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.745408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.745457 (+ 49us) server_negotiation.cc:407] Connection header received
0923 15:11:50.745528 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.745529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.745537 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.745588 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.746018 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.746564 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.747618 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.747823 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.747887 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.748889 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.748891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.748891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.748897 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.750379 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.750752 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.750753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.750753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.750803 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.750989 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.750989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.750989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.751078 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.751137 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.751295 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.751354 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:50.751473 29001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.745267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:39746)
0923 15:11:50.745390 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:50.745400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:50.745467 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:50.745594 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:50.745595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:50.745813 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:50.745951 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.745953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.746643 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.746644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.747512 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:50.747515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.747909 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.747910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:50.747978 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.748405 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:50.748410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:50.748797 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:50.750456 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.750458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.750458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.750677 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.750866 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:50.750867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:50.750867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:50.750930 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:50.751137 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:50.751138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:50.751184 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 15:11:50.751267 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 15:11:50.751371 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":86,"threads_started":1}
I20260923 15:11:50.752046 28451 tablet_copy_service.cc:140] P 03997f227c114b958667f054eaa101f6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eb48151fe57d4268b99e218f1bcfd2ba ({username='oryx', principal='oryx/127.21.204.125@KRBTEST.COM'} at 127.0.0.1:39746)
I20260923 15:11:50.752159 28451 tablet_copy_service.cc:161] P 03997f227c114b958667f054eaa101f6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eb48151fe57d4268b99e218f1bcfd2ba at {username='oryx', principal='oryx/127.21.204.125@KRBTEST.COM'} at 127.0.0.1:39746: session id = eb48151fe57d4268b99e218f1bcfd2ba-00000000000000000000000000000000
I20260923 15:11:50.752832 28451 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 15:11:50.753471 28895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 15:11:50.754474 28895 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: tablet copy: Starting download of 0 data blocks...
I20260923 15:11:50.754654 28895 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: tablet copy: Starting download of 1 WAL segments...
I20260923 15:11:50.755708 28895 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 15:11:50.756484 28451 tablet_copy_service.cc:342] P 03997f227c114b958667f054eaa101f6: Request end of tablet copy session eb48151fe57d4268b99e218f1bcfd2ba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.204.125@KRBTEST.COM'} at 127.0.0.1:39746
I20260923 15:11:50.756563 28451 tablet_copy_service.cc:433] P 03997f227c114b958667f054eaa101f6: ending tablet copy session eb48151fe57d4268b99e218f1bcfd2ba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eb48151fe57d4268b99e218f1bcfd2ba
I20260923 15:11:50.759022 28895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:50.760159 29002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.760264 29003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.762714 29005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.762925 28895 server_base.cc:1034] running on GCE node
I20260923 15:11:50.763039 28895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:50.763175 28895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:50.764304 28895 hybrid_clock.cc:648] HybridClock initialized: now 1790176310764295 us; error 24 us; skew 500 ppm
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:50.765628 28895 init.cc:377] Logged in from keytab as oryx/127.21.204.125@KRBTEST.COM (short username oryx)
I20260923 15:11:50.765947 28895 webserver.cc:460] Webserver started at http://127.21.204.125:45871/ using document root <none> and password file <none>
I20260923 15:11:50.766086 28895 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:50.766129 28895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:50.766721 28895 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:11:50.767282 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.767422 28895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:11:50.767469 28895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "e5c7ca50c5d61c0b5b7d4539b551828e"
server_key_iv: "ae710c60b774f780b2d9049268156ea3"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.767526 28895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:50.795083 28895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:50.796948 28895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:50.800484 28895 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.125:39959
I20260923 15:11:50.800506 29064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.125:39959 every 8 connection(s)
I20260923 15:11:50.800849 28895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
I20260923 15:11:50.801743 29065 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:11:50.802264 29065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Bootstrap starting.
I20260923 15:11:50.805182 29065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:11:50.805402 29065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: Bootstrap complete.
I20260923 15:11:50.805627 29065 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } }
I20260923 15:11:50.805722 29065 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb48151fe57d4268b99e218f1bcfd2ba, State: Initialized, Role: LEARNER
I20260923 15:11:50.805768 29065 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } }
I20260923 15:11:50.805953 29065 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:50.805984 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } }
I20260923 15:11:50.806133 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: LEARNER
I20260923 15:11:50.807471 29079 catalog_manager.cc:1259] Loaded cluster ID: d1f82c282da24552925d540452d15f68
I20260923 15:11:50.807520 29079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: loading cluster ID for follower catalog manager: success
I20260923 15:11:50.808741 29079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: acquiring CA information for follower catalog manager: success
I20260923 15:11:50.809054 29079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 15:11:50.851563 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:50.845620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.21.204.125:39959)
0923 15:11:50.845771 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:50.845773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:50.845782 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:50.845807 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:50.845808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:50.845819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:50.845945 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:50.846477 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.847148 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.847863 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:50.848066 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:50.848130 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:50.849332 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:50.849333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:50.849333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:50.849340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:50.850649 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.850975 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.850975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.850976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.851027 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:50.851187 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:50.851187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:50.851188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:50.851286 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:50.851357 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:50.851436 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 15:11:50.851495 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.21.204.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.124.keytab.
Entry for principal oryx/127.21.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.124.keytab.
I20260923 15:11:50.876837 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:43357
--webserver_interface=127.21.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.124.keytab
--principal=oryx/127.21.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:50.962096 29083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:50.962298 29083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:50.962321 29083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:50.963431 29083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:50.963476 29083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:50.963493 29083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:50.963507 29083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:50.963521 29083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:50.964974 29083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.124.keytab
--principal=oryx/127.21.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:43357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:50.965190 29083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:50.965369 29083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:50.973107 29088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.973155 29089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:50.973171 29091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:50.973820 29083 server_base.cc:1034] running on GCE node
I20260923 15:11:50.974134 29083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:50.974496 29083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:50.975651 29083 hybrid_clock.cc:648] HybridClock initialized: now 1790176310975640 us; error 30 us; skew 500 ppm
Sep 23 15:11:50 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:50.978495 29083 init.cc:377] Logged in from keytab as oryx/127.21.204.124@KRBTEST.COM (short username oryx)
I20260923 15:11:50.978961 29083 webserver.cc:460] Webserver started at http://127.21.204.124:40643/ using document root <none> and password file <none>
I20260923 15:11:50.979157 29083 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:50.979198 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:50.979290 29083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:50.980302 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/instance:
uuid: "d1c46dc772ee47fca057d6ee5c5586d7"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "4cbca0eddf21706f473e23d51668d5ca"
server_key_iv: "cc6f566412ae802c5574950f153a2db8"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.980603 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal/instance:
uuid: "d1c46dc772ee47fca057d6ee5c5586d7"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "4cbca0eddf21706f473e23d51668d5ca"
server_key_iv: "cc6f566412ae802c5574950f153a2db8"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.981966 29083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:11:50.982676 29098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:50.982980 29083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:50.983088 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
uuid: "d1c46dc772ee47fca057d6ee5c5586d7"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "4cbca0eddf21706f473e23d51668d5ca"
server_key_iv: "cc6f566412ae802c5574950f153a2db8"
server_key_version: "encryptionkey@0"
I20260923 15:11:50.983174 29083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:51.000250 29083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:51.002485 29083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:51.002645 29083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:51.007889 29083 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.124:43357
I20260923 15:11:51.007937 29150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.124:43357 every 8 connection(s)
I20260923 15:11:51.008287 29083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
I20260923 15:11:51.009270 29151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:51.010877 29151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:11:51.011852 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 29083
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.124@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.204.124@KRBTEST.COM
I20260923 15:11:51.021953 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.011925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.21.204.126:40489)
0923 15:11:51.011976 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.011978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.012145 (+ 167us) server_negotiation.cc:407] Connection header received
0923 15:11:51.012728 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.012730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.012740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.012803 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.013583 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.014419 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.015719 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.015984 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.016061 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.018653 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.018654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.018655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.018664 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.020385 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.020942 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.020943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.020943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.020995 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.021283 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.021284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.021285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.021403 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.021479 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.021754 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.021834 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 15:11:51.022150 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.011763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59092)
0923 15:11:51.011993 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.012048 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.012632 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.012858 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.012861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.013226 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.013461 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.013467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.014527 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.014528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.015618 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.015621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.016103 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.016104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.016205 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.016818 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.016830 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.018550 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.020471 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.020476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.020479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.020862 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.021079 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.021080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.021080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.021202 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.021470 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.021476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.021615 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.021729 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.021881 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":120,"threads_started":1}
I20260923 15:11:51.022271 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.012414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.21.204.124:43357)
0923 15:11:51.012589 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.012594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.012626 (+ 32us) server_negotiation.cc:407] Connection header received
0923 15:11:51.012723 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.012725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.012746 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.012832 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.013634 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.014876 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.015887 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.016304 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.016362 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.019157 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.019172 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.019172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.019178 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.020963 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.021307 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.021310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.021310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.021369 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.021601 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.021602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.021602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.021754 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.021917 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.022027 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.022112 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260923 15:11:51.023031 29151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.124@KRBTEST.COM for oryx/127.21.204.125@KRBTEST.COM
I20260923 15:11:51.029529 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.023277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.0.0.1:40348)
0923 15:11:51.023324 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.023334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.023369 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.023513 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.023514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.023641 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.023691 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.023692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.024345 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.024346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.024853 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.024854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.025253 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.025254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.025324 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.025734 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.025739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.027008 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.028494 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.028495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.028495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.028666 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.028890 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.028891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.028891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.028949 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.029203 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.029204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.029249 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.029334 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.029434 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 15:11:51.029532 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.023348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.21.204.125:39959)
0923 15:11:51.023400 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.023402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.023412 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:51.023438 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.023439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.023452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.023505 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.023768 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.024263 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.024936 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.025168 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.025251 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.027098 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.027100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.027100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.027108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.028381 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.028759 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.028760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.028761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.028822 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.029017 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.029018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.029018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.029136 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.029212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.029368 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.029448 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:11:51.030367 29151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176310, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.124@KRBTEST.COM for oryx/127.21.204.124@KRBTEST.COM
I20260923 15:11:51.036207 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.030611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.0.0.1:36148)
0923 15:11:51.030657 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.030667 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.030709 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.030833 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.030833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.030937 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.030965 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.030965 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.031577 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.031577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.032037 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.032038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.032405 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.032406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.032442 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.032755 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.032759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.033992 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.035458 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.035458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.035459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.035600 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.035772 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.035773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.035773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.035822 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.035988 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.035989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.036019 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.036064 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.036133 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 15:11:51.036307 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.030674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.21.204.124:43357)
0923 15:11:51.030714 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.030716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.030739 (+ 23us) server_negotiation.cc:407] Connection header received
0923 15:11:51.030768 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.030769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.030778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.030828 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.031024 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.031465 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.032127 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.032336 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.032399 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.034052 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.034054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.034054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.034061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.035388 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.035666 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.035666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.035667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.035712 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.035860 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.035860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.035860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.035941 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.035983 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.036088 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.036175 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":18}
I20260923 15:11:51.037698 29151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Bootstrap starting.
I20260923 15:11:51.038648 29151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:51.038947 29151 log.cc:824] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:51.039659 29151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: No bootstrap required, opened a new log
I20260923 15:11:51.041131 29151 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } }
I20260923 15:11:51.041280 29151 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:51.041337 29151 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c46dc772ee47fca057d6ee5c5586d7, State: Initialized, Role: FOLLOWER
I20260923 15:11:51.041493 29151 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } }
I20260923 15:11:51.041822 29151 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:51.041805 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } } }
I20260923 15:11:51.041977 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 15:11:51.043548 29167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:11:51.043613 29167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:11:51.046226 29083 master_runner.cc:428] Detected that this master d1c46dc772ee47fca057d6ee5c5586d7 is joining an existing cluster
I20260923 15:11:51.046265 29083 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.204.124:43357
I20260923 15:11:51.054443 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.047578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59094)
0923 15:11:51.047779 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.047790 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.047879 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.048074 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.048075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.048304 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.048389 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.048391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.049368 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.049369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.050299 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.050301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.050808 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.050866 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.051302 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.051306 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.051618 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.053412 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.053413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.053413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.053577 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.053804 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.053805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.053805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.053857 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.054197 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.054198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.054243 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.054299 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.054372 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":3,"thread_start_us":148,"threads_started":1}
I20260923 15:11:51.054520 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.047550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.21.204.126:40489)
0923 15:11:51.047596 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.047599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.047840 (+ 241us) server_negotiation.cc:407] Connection header received
0923 15:11:51.047958 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.047960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.047973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.048007 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.048477 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.049258 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050393 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.050680 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050717 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.051716 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.051718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.051718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.051726 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.053318 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.053653 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.053655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.053655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.053726 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.053984 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.053985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.053985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.054126 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.054211 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.054346 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.054426 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176309, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.126@KRBTEST.COM for oryx/127.21.204.124@KRBTEST.COM
I20260923 15:11:51.073803 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 28493
I20260923 15:11:51.074569 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.047906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.21.204.125:39959)
0923 15:11:51.047947 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.047948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.048164 (+ 216us) server_negotiation.cc:407] Connection header received
0923 15:11:51.048255 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.048255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.048265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.048294 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.048745 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.049455 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050540 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.050840 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050878 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.071088 (+ 20210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.071092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.071092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.071101 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.072853 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.073574 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.073575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.073576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.073639 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.073860 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.073861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.073862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.074024 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.074064 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.074334 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.074425 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:11:51.075178 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.047988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.0.0.1:40360)
0923 15:11:51.048111 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.048120 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.048186 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.048360 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.048361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.048600 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.048649 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.048650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.049549 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.049549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.050461 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.050463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.050954 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.051018 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.070536 (+ 19518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.070543 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.070969 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.072953 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.072954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.072955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.073483 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.073715 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.073715 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.073716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.073783 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.074166 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.074166 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.074223 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.074915 (+ 692us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.075039 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":45,"thread_start_us":70,"threads_started":1}
I20260923 15:11:51.075644 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.056732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.0.0.1:36176)
0923 15:11:51.056778 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.056790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.056840 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.057181 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.057182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.057308 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.057345 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.057346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.058331 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.058333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.058843 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.058844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.059320 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.059321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.059369 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.059759 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.059785 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.061175 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.073672 (+ 12497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.073673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.073674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.073929 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.074961 (+ 1032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.074962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.074962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.075028 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.075328 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.075329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.075375 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.075427 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.075547 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 15:11:51.076248 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.056846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.21.204.124:43357)
0923 15:11:51.057024 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.057026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.057034 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:51.057080 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.057081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.057092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.057166 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.057419 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.058236 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.058921 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.059234 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.059307 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.061283 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.061285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.061285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.061293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.073575 (+ 12282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.074829 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.074830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.074830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.074883 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.075099 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.075100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.075100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.075222 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.076048 (+ 826us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.076093 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.076171 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260923 15:11:51.079645 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.047954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.21.204.124:43357)
0923 15:11:51.047990 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.047992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.048096 (+ 104us) server_negotiation.cc:407] Connection header received
0923 15:11:51.048190 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.048191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.048199 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.048242 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.048768 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.049424 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050582 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.050835 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050876 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.071812 (+ 20936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.071813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.071814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.071821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.078488 (+ 6667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.078853 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.078854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.078855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.078909 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.079105 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.079106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.079106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.079208 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.079285 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.079445 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.079544 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":18}
I20260923 15:11:51.079751 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.047781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.0.0.1:36162)
0923 15:11:51.048038 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.048048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.048117 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.048303 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.048304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.048607 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.048676 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.048678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.049522 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.049523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.050469 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.050472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.050952 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.050953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.051013 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.071214 (+ 20201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.071220 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.071698 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.078592 (+ 6894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.078593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.078594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.078762 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.078976 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.078977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.078977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.079033 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.079286 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.079287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.079338 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.079423 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.079609 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":141,"thread_start_us":100,"threads_started":1}
I20260923 15:11:51.081765 28431 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.204.124:43357: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7"
member_type: NON_VOTER
last_known_addr {
host: "127.21.204.124"
port: 43357
}
attrs {
promote: true
}
}
dest_uuid: "03997f227c114b958667f054eaa101f6"
cas_config_opid_index: 5
I20260923 15:11:51.082058 28431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } }
I20260923 15:11:51.082329 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:44703
--local_ip_for_outbound_sockets=127.21.204.65
--tserver_master_addrs=127.21.204.126:40489
--webserver_port=36207
--webserver_interface=127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab
--principal=oryx/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:11:51.082994 29125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:51.084007 29125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Refusing update from remote peer 03997f227c114b958667f054eaa101f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 15:11:51.083956 28990 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.084278 28991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:51.085114 28990 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.204.124:43357
I20260923 15:11:51.085194 28990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.085279 28990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.085562 29125 raft_consensus.cc:858] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } }
I20260923 15:11:51.085598 29083 master.cc:537] Master@127.21.204.124:43357 shutting down...
W20260923 15:11:51.085635 29125 raft_consensus.cc:1565] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } }
I20260923 15:11:51.086175 29125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } }
I20260923 15:11:51.087376 29181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } }
I20260923 15:11:51.087486 29181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 15:11:51.087509 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03997f227c114b958667f054eaa101f6. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } }
I20260923 15:11:51.087580 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 15:11:51.087519 29181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } }
I20260923 15:11:51.087613 29181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 15:11:51.088040 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.082581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.0.0.1:40376)
0923 15:11:51.082686 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.082697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.082740 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.082961 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.082962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.082964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:51.083012 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.083014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.083999 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.084000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.085164 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.085167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.087749 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.087750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.087835 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.087859 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.087918 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.087967 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 15:11:51.088451 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.082687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.21.204.125:39959)
0923 15:11:51.082739 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.082741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.082749 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:51.082806 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.082806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.082831 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.082900 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:51.083133 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.083887 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.086412 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.087060 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.087135 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.087222 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.088330 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.088378 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 15:11:51.088419 28991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.088495 28991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.088662 29039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 LEARNER]: Refusing update from remote peer 03997f227c114b958667f054eaa101f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 15:11:51.088919 28990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260923 15:11:51.089087 28417 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 -> Peer d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124:43357): Couldn't send request to peer d1c46dc772ee47fca057d6ee5c5586d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 15:11:51.089490 29039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.090837 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03997f227c114b958667f054eaa101f6. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.090927 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: LEARNER
I20260923 15:11:51.090965 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: NON_VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: true } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.091008 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: LEARNER
I20260923 15:11:51.091172 28990 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: attempting to promote NON_VOTER eb48151fe57d4268b99e218f1bcfd2ba to VOTER
I20260923 15:11:51.091349 28990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } }
I20260923 15:11:51.091790 29039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 LEARNER]: Refusing update from remote peer 03997f227c114b958667f054eaa101f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 15:11:51.092087 28991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260923 15:11:51.092162 28417 consensus_peers.cc:487] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 -> Peer d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124:43357): Couldn't send request to peer d1c46dc772ee47fca057d6ee5c5586d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 15:11:51.092482 29178 mvcc.cc:205] Tried to move back new op lower bound from 7332562165887778816 to 7332562165817290752. Current Snapshot: MvccSnapshot[applied={T|T < 7332562165828878336}]
I20260923 15:11:51.092543 29083 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260923 15:11:51.092607 29083 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260923 15:11:51.092634 29083 tablet_replica.cc:330] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: stopping tablet replica
I20260923 15:11:51.093427 28991 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.094435 28991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.094452 29188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.094519 28991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.094527 29188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.094993 29039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.095937 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.096021 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:11:51.107616 29083 master.cc:552] Master@127.21.204.124:43357 shutdown complete.
I20260923 15:11:51.107693 29083 master_runner.cc:305] Clearing existing system tablet
I20260923 15:11:51.107829 29083 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:51.107892 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:51.108529 29083 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:51.109033 29190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.109206 29083 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:51.109272 29083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.109314 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
uuid: "d1c46dc772ee47fca057d6ee5c5586d7"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "4cbca0eddf21706f473e23d51668d5ca"
server_key_iv: "cc6f566412ae802c5574950f153a2db8"
server_key_version: "encryptionkey@0"
I20260923 15:11:51.110059 29083 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:51.111441 29083 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 15:11:51.111524 29083 log.cc:1196] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 15:11:51.111743 29083 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Deleting consensus metadata
I20260923 15:11:51.111886 29083 master_runner.cc:315] Copying system tablet from 127.21.204.126:40489
I20260923 15:11:51.112511 29083 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: tablet copy: Beginning tablet copy session from remote peer at address 127.21.204.126:40489
I20260923 15:11:51.119180 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.112882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.21.204.126:40489)
0923 15:11:51.112929 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.112931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.113052 (+ 121us) server_negotiation.cc:407] Connection header received
0923 15:11:51.113132 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.113133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.113144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.113185 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.113518 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.114202 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.115219 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.115428 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.115525 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.116500 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.116502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.116502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.116509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.118027 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.118389 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.118390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.118390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.118452 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.118652 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.118653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.118653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.118772 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.118844 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.119004 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.119079 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:11:51.119206 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.112843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59102)
0923 15:11:51.112972 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.112983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.113068 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.113199 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.113200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.113387 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.113434 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.113436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.114285 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.114286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.115126 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.115128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.115513 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.115514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.115578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.116080 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.116085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.116390 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.118123 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.118124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.118125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.118306 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.118521 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.118522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.118522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.118579 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.118845 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.118847 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.118891 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.118975 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.119081 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260923 15:11:51.119757 28451 tablet_copy_service.cc:140] P 03997f227c114b958667f054eaa101f6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d1c46dc772ee47fca057d6ee5c5586d7 ({username='oryx', principal='oryx/127.21.204.124@KRBTEST.COM'} at 127.0.0.1:59102)
I20260923 15:11:51.119843 28451 tablet_copy_service.cc:161] P 03997f227c114b958667f054eaa101f6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d1c46dc772ee47fca057d6ee5c5586d7 at {username='oryx', principal='oryx/127.21.204.124@KRBTEST.COM'} at 127.0.0.1:59102: session id = d1c46dc772ee47fca057d6ee5c5586d7-00000000000000000000000000000000
I20260923 15:11:51.120302 28451 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 15:11:51.120771 29083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 15:11:51.121649 29083 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: tablet copy: Starting download of 0 data blocks...
I20260923 15:11:51.121807 29083 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: tablet copy: Starting download of 1 WAL segments...
I20260923 15:11:51.122488 29083 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 15:11:51.123200 28451 tablet_copy_service.cc:342] P 03997f227c114b958667f054eaa101f6: Request end of tablet copy session d1c46dc772ee47fca057d6ee5c5586d7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.204.124@KRBTEST.COM'} at 127.0.0.1:59102
I20260923 15:11:51.123253 28451 tablet_copy_service.cc:433] P 03997f227c114b958667f054eaa101f6: ending tablet copy session d1c46dc772ee47fca057d6ee5c5586d7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d1c46dc772ee47fca057d6ee5c5586d7
I20260923 15:11:51.125636 29083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:51.126983 29197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:51.127095 29200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:51.127336 29083 server_base.cc:1034] running on GCE node
W20260923 15:11:51.130016 29198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:51.130177 29083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:51.130328 29083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:51.131445 29083 hybrid_clock.cc:648] HybridClock initialized: now 1790176311131437 us; error 23 us; skew 500 ppm
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:51.132776 29083 init.cc:377] Logged in from keytab as oryx/127.21.204.124@KRBTEST.COM (short username oryx)
I20260923 15:11:51.133068 29083 webserver.cc:460] Webserver started at http://127.21.204.124:38099/ using document root <none> and password file <none>
I20260923 15:11:51.133185 29083 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:51.133220 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:51.133827 29083 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:11:51.134395 29207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.134546 29083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.134593 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
uuid: "d1c46dc772ee47fca057d6ee5c5586d7"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "4cbca0eddf21706f473e23d51668d5ca"
server_key_iv: "cc6f566412ae802c5574950f153a2db8"
server_key_version: "encryptionkey@0"
I20260923 15:11:51.134652 29083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:51.143047 29083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:51.144781 29083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:51.148033 29083 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.124:43357
I20260923 15:11:51.148463 29083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
I20260923 15:11:51.148866 29259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.124:43357 every 8 connection(s)
I20260923 15:11:51.149495 29260 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:11:51.149955 29260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Bootstrap starting.
I20260923 15:11:51.151784 29260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:11:51.152005 29260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Bootstrap complete.
I20260923 15:11:51.152211 29260 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } }
I20260923 15:11:51.152302 29260 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1c46dc772ee47fca057d6ee5c5586d7, State: Initialized, Role: LEARNER
I20260923 15:11:51.152348 29260 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } }
I20260923 15:11:51.152527 29260 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:51.152519 29263 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.152729 29263 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: LEARNER
I20260923 15:11:51.154359 29274 catalog_manager.cc:1259] Loaded cluster ID: d1f82c282da24552925d540452d15f68
I20260923 15:11:51.154443 29274 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: loading cluster ID for follower catalog manager: success
I20260923 15:11:51.155334 29274 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: acquiring CA information for follower catalog manager: success
I20260923 15:11:51.155616 29274 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 15:11:51.208432 29177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:51.208781 29177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:51.208823 29177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:51.210716 29177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:51.210790 29177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:51.210835 29177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:51.213224 29177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab
--principal=oryx/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:44703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=36207
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:51.213501 29177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:51.213963 29177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:51.219873 29279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:51.219986 29280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:51.219918 29282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:51.220166 29177 server_base.cc:1034] running on GCE node
I20260923 15:11:51.220369 29177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:51.220618 29177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:51.221802 29177 hybrid_clock.cc:648] HybridClock initialized: now 1790176311221778 us; error 43 us; skew 500 ppm
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:51.224629 29177 init.cc:377] Logged in from keytab as oryx/127.21.204.65@KRBTEST.COM (short username oryx)
I20260923 15:11:51.225109 29177 webserver.cc:460] Webserver started at http://127.21.204.65:36207/ using document root <none> and password file <none>
I20260923 15:11:51.225363 29177 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:51.225455 29177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:51.226934 29177 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:51.227672 29289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.227885 29177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:51.227988 29177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "4ce281b9d1e4434cb2a07ed3f186cb24"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "2d5707aadfb69c74b14966469d4b6f73"
server_key_iv: "1478528aca4453081d63a43b7ea5c274"
server_key_version: "encryptionkey@0"
I20260923 15:11:51.228346 29177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:51.244170 29177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:51.246922 29177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:51.247092 29177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:51.247352 29177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:51.247673 29177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:51.247728 29177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.247771 29177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:51.247797 29177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.255189 29177 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:44703
I20260923 15:11:51.255237 29402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:44703 every 8 connection(s)
I20260923 15:11:51.255625 29177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:51.258939 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 29177
I20260923 15:11:51.259101 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 28627
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.125@KRBTEST.COM
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.124@KRBTEST.COM
I20260923 15:11:51.266371 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:42863
--local_ip_for_outbound_sockets=127.21.204.66
--tserver_master_addrs=127.21.204.126:40489
--webserver_port=43155
--webserver_interface=127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab
--principal=oryx/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:11:51.269155 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.256863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:33743 (local address 127.21.204.126:40489)
0923 15:11:51.256911 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.256913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.257128 (+ 215us) server_negotiation.cc:407] Connection header received
0923 15:11:51.258117 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.258119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.258132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.258165 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.259123 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.260096 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.261523 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.261817 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.261855 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.265551 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.265552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.265553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.265561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.267366 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.268166 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.268167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.268168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.268246 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.268441 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.268442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.268442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.268580 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.268624 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.268953 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.269047 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:11:51.269177 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.257240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:60073 (local address 127.21.204.125:39959)
0923 15:11:51.257286 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.257288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.257330 (+ 42us) server_negotiation.cc:407] Connection header received
0923 15:11:51.258089 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.258090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.258103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.258156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.259107 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.259885 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.261539 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.261917 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.261961 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.265543 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.265544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.265544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.265552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.267507 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.267995 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.267996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.267997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.268058 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.268356 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.268356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.268357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.268508 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.268552 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.268954 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.269037 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:11:51.269906 29409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.256760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:33743)
0923 15:11:51.257055 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.257066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.258018 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.258241 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.258245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.258748 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.259006 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.259014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.260211 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.260212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.261432 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.261435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.261963 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.261963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.262094 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.262755 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.262760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.265436 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.267852 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.267853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.267854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.268064 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.268316 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.268317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.268317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.268372 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.268670 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.268671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.268789 (+ 118us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.269326 (+ 537us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.269646 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":149,"thread_start_us":106,"threads_started":1}
I20260923 15:11:51.270074 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.257154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.21.204.65:60073)
0923 15:11:51.257267 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.257277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.257998 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.258240 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.258244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.258716 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.259004 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.259009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.260017 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.260018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.261434 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.261438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.262037 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.262038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.262113 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.262725 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.262736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.265434 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.267569 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.267576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.267579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.267897 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.268135 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.268136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.268136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.268265 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.268615 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.268623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.268789 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.269328 (+ 539us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.269556 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":55,"thread_start_us":67,"threads_started":1}
I20260923 15:11:51.270414 29405 heartbeater.cc:347] Connected to a master server at 127.21.204.125:39959
I20260923 15:11:51.270517 29405 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.270665 29405 heartbeater.cc:511] Master 127.21.204.125:39959 requested a full tablet report, sending...
I20260923 15:11:51.271049 29404 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40489
I20260923 15:11:51.271114 29404 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.271188 29404 heartbeater.cc:511] Master 127.21.204.126:40489 requested a full tablet report, sending...
I20260923 15:11:51.271291 29029 ts_manager.cc:194] Registered new tserver with Master: 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703)
I20260923 15:11:51.271449 28431 ts_manager.cc:194] Re-registered known tserver with Master: 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703)
I20260923 15:11:51.271970 28431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:33743
I20260923 15:11:51.273864 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.256430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.21.204.65:49895)
0923 15:11:51.256594 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.256629 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.257976 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.258273 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.258274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.260982 (+ 2708us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.261043 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.261045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.262210 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.262211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.263461 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.263464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.264013 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.264020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.264086 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.264648 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.264654 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.268502 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.272811 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.272812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.272813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.272985 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.273207 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.273208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.273209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.273268 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.273516 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.273517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.273559 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.273647 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.273757 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260923 15:11:51.273854 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.256412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:49895 (local address 127.21.204.124:43357)
0923 15:11:51.256542 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.256544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.256717 (+ 173us) server_negotiation.cc:407] Connection header received
0923 15:11:51.258154 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.258155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.258168 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.258231 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.261139 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.262123 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.263563 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.263872 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.263951 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.268597 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.268599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.268599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.268606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.272707 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.273064 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.273065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.273065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.273125 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.273332 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.273333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.273334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.273447 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.273524 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.273672 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.273761 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260923 15:11:51.274255 29403 heartbeater.cc:347] Connected to a master server at 127.21.204.124:43357
I20260923 15:11:51.274304 29403 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.274384 29403 heartbeater.cc:511] Master 127.21.204.124:43357 requested a full tablet report, sending...
I20260923 15:11:51.274798 29218 ts_manager.cc:194] Registered new tserver with Master: 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703)
W20260923 15:11:51.353137 29411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:51.353346 29411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:51.353372 29411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:51.354630 29411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:51.354677 29411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:51.354707 29411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:51.356138 29411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab
--principal=oryx/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:42863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=43155
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:51.356325 29411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:51.356513 29411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:51.361984 29416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:51.361979 29419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:51.362043 29417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:51.362154 29411 server_base.cc:1034] running on GCE node
I20260923 15:11:51.362393 29411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:51.362599 29411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:51.363747 29411 hybrid_clock.cc:648] HybridClock initialized: now 1790176311363723 us; error 36 us; skew 500 ppm
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:51.366334 29411 init.cc:377] Logged in from keytab as oryx/127.21.204.66@KRBTEST.COM (short username oryx)
I20260923 15:11:51.366947 29411 webserver.cc:460] Webserver started at http://127.21.204.66:43155/ using document root <none> and password file <none>
I20260923 15:11:51.367280 29411 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:51.367342 29411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:51.368716 29411 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:51.369374 29426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.369638 29411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:51.369727 29411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "9b6dc5f13a1341dc9fc047f5f112bb74"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "5e0c235223406b6e89a2f06f84281b8f"
server_key_iv: "a0e43d1df065b9d707574bac841b6564"
server_key_version: "encryptionkey@0"
I20260923 15:11:51.370157 29411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:51.391116 29411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:51.393236 29411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:51.393392 29411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:51.393663 29411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:51.394018 29411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:51.394054 29411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.394078 29411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:51.394126 29411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.400044 29411 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:42863
I20260923 15:11:51.400146 29539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:42863 every 8 connection(s)
I20260923 15:11:51.400465 29411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:51.402436 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 29411
I20260923 15:11:51.402616 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 28761
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.124@KRBTEST.COM
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.125@KRBTEST.COM
I20260923 15:11:51.410614 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:34017
--local_ip_for_outbound_sockets=127.21.204.67
--tserver_master_addrs=127.21.204.126:40489
--webserver_port=37691
--webserver_interface=127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab
--principal=oryx/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:11:51.413831 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.401608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:51839 (local address 127.21.204.125:39959)
0923 15:11:51.401693 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.401695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.401747 (+ 52us) server_negotiation.cc:407] Connection header received
0923 15:11:51.403514 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.403515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.403529 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.403559 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.403913 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.404688 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406391 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.406680 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406720 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.410806 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.410807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.410807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.410815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.412516 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.412984 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.412985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.412986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.413043 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.413245 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.413246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.413246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.413381 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.413431 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.413660 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.413741 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:11:51.413897 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.401225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:52991 (local address 127.21.204.124:43357)
0923 15:11:51.401289 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.401292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.401540 (+ 248us) server_negotiation.cc:407] Connection header received
0923 15:11:51.402792 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.402793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.402810 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.402841 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.403919 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.404742 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406381 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.406763 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406806 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.409699 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.409701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.409701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.409710 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.412213 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.412714 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.412716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.412716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.412780 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.413062 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.413063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.413064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.413210 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.413260 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.413663 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.413764 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 15:11:51.414798 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.401217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.21.204.66:52991)
0923 15:11:51.401398 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.401442 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.402681 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.403029 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.403033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.403479 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.403807 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.403813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.404840 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.404840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.406282 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.406285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.406884 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.406961 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.407877 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.407883 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.409595 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.412310 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.412317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.412319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.412621 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.412856 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.412857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.412857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.412989 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.413319 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.413325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.413483 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.414505 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.414634 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":80,"thread_start_us":109,"threads_started":1}
I20260923 15:11:51.415539 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.401545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.21.204.66:51839)
0923 15:11:51.401683 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.401694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.402627 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.403624 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.403625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.403763 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.403812 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.403813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.404825 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.404826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.406287 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.406289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.406813 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.406984 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.407754 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.407759 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.410699 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.412629 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.412630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.412631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.412895 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.413109 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.413110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.413110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.413172 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.413466 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.413467 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.413507 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.414034 (+ 527us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.414225 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":109,"thread_start_us":72,"threads_started":1}
I20260923 15:11:51.415635 29540 heartbeater.cc:347] Connected to a master server at 127.21.204.124:43357
I20260923 15:11:51.415752 29540 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.415946 29540 heartbeater.cc:511] Master 127.21.204.124:43357 requested a full tablet report, sending...
I20260923 15:11:51.416082 29542 heartbeater.cc:347] Connected to a master server at 127.21.204.125:39959
I20260923 15:11:51.416132 29542 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.416208 29542 heartbeater.cc:511] Master 127.21.204.125:39959 requested a full tablet report, sending...
I20260923 15:11:51.416625 29029 ts_manager.cc:194] Registered new tserver with Master: 9b6dc5f13a1341dc9fc047f5f112bb74 (127.21.204.66:42863)
I20260923 15:11:51.416643 29218 ts_manager.cc:194] Registered new tserver with Master: 9b6dc5f13a1341dc9fc047f5f112bb74 (127.21.204.66:42863)
I20260923 15:11:51.417191 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.401268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:42839 (local address 127.21.204.126:40489)
0923 15:11:51.401318 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.401320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.401547 (+ 227us) server_negotiation.cc:407] Connection header received
0923 15:11:51.403310 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.403311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.403325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.403355 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.403914 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.404684 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406391 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.406722 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406761 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.410188 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.410189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.410190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.410197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.412605 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.414628 (+ 2023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.414630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.414630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.414693 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.415050 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.415051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.415052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.415201 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.415252 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.416955 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.417052 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:51.417564 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.401220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:42839)
0923 15:11:51.401406 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.401442 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.402619 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.403419 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.403420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.403635 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.403807 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.403813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.404825 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.404826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.406287 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.406289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.406875 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.406876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.406951 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.407677 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.407686 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.410078 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.413793 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.413794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.413794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.414529 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.414767 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.414768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.414768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.414831 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.415400 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.415401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.415444 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.417298 (+ 1854us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.417413 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":88,"spinlock_wait_cycles":10496,"thread_start_us":119,"threads_started":1}
I20260923 15:11:51.418025 29541 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40489
I20260923 15:11:51.418089 29541 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.418171 29541 heartbeater.cc:511] Master 127.21.204.126:40489 requested a full tablet report, sending...
I20260923 15:11:51.418426 28431 ts_manager.cc:194] Re-registered known tserver with Master: 9b6dc5f13a1341dc9fc047f5f112bb74 (127.21.204.66:42863)
I20260923 15:11:51.418967 28431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:42839
W20260923 15:11:51.496476 29547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:51.496688 29547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:51.496712 29547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:51.497804 29547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:51.497840 29547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:51.497866 29547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:51.499300 29547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab
--principal=oryx/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:34017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=37691
--tserver_master_addrs=127.21.204.126:40489,127.21.204.125:39959,127.21.204.124:43357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:51.499483 29547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:51.499670 29547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:51.504681 29555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:51.504721 29552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:51.504781 29553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:51.504925 29547 server_base.cc:1034] running on GCE node
I20260923 15:11:51.505204 29547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:51.505441 29547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:51.506621 29547 hybrid_clock.cc:648] HybridClock initialized: now 1790176311506616 us; error 38 us; skew 500 ppm
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:51.509002 29547 init.cc:377] Logged in from keytab as oryx/127.21.204.67@KRBTEST.COM (short username oryx)
I20260923 15:11:51.509428 29547 webserver.cc:460] Webserver started at http://127.21.204.67:37691/ using document root <none> and password file <none>
I20260923 15:11:51.509665 29547 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:51.509748 29547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:51.511157 29547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:51.511857 29562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.512048 29547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:51.512156 29547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "9d14836f0e77457ca7a893e002d211da"
format_stamp: "Formatted at 2026-09-23 15:11:50 on dist-test-slave-t5lt"
server_key: "c13b0db34f2cc258988d7f5b55c9ca04"
server_key_iv: "359dec5571a943861b5f44d6fabac3c2"
server_key_version: "encryptionkey@0"
I20260923 15:11:51.512488 29547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:51.524703 29547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:51.526850 29547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:51.527005 29547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:51.527227 29547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:11:51.527539 29547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:51.527593 29547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.527628 29547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:51.527658 29547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:51.533602 29547 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:34017
I20260923 15:11:51.533739 29675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:34017 every 8 connection(s)
I20260923 15:11:51.534083 29547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:51.537990 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 29547
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.124@KRBTEST.COM
Sep 23 15:11:51 dist-test-slave-t5lt krb5kdc[28386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176311, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.125@KRBTEST.COM
I20260923 15:11:51.547497 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.534875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:41165 (local address 127.21.204.124:43357)
0923 15:11:51.534925 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.534927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.535235 (+ 308us) server_negotiation.cc:407] Connection header received
0923 15:11:51.536395 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.536397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.536409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.536464 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.537327 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.538087 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.539499 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.539796 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.539863 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.543882 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.543883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.543883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.543890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.545697 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.546347 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.546349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.546350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.546409 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.546718 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.546719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.546720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.546860 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.546904 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.547304 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.547391 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:11:51.548051 29080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.535214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:45321 (local address 127.21.204.125:39959)
0923 15:11:51.535263 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.535267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.536613 (+ 1346us) server_negotiation.cc:407] Connection header received
0923 15:11:51.536655 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.536657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.536669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.536719 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.537383 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.537953 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.539402 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.539697 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.539759 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.544121 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.544123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.544123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.544129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.546770 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.547178 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.547179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.547180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.547233 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.547448 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.547449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.547449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.547580 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.547626 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.547878 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.547948 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:51.548094 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.534843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:42949)
0923 15:11:51.535301 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.535309 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.536315 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.536602 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.536603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.536964 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.537161 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.537165 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.538176 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.538176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.539372 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.539374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.540319 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.540320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.540385 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.540792 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.540796 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.543653 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.546533 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.546533 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.546534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.546732 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.547203 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.547204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.547204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.547269 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.547541 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.547542 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.547583 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.547660 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.547868 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":99,"thread_start_us":93,"threads_started":1}
I20260923 15:11:51.549319 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.535115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:42949 (local address 127.21.204.126:40489)
0923 15:11:51.535169 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.535171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.535373 (+ 202us) server_negotiation.cc:407] Connection header received
0923 15:11:51.536446 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.536448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.536462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.536511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.537254 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.538076 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.539911 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.540221 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.540311 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.543735 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.543736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.543737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.543743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.546454 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.547051 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.547053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.547053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.547113 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.547336 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.547337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.547337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.547456 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.547513 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.549142 (+ 1629us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.549218 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:11:51.549738 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.535141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.21.204.67:45321)
0923 15:11:51.536528 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.536539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.536600 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.536722 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.536722 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.537043 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.537286 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.537288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.538093 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.538094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.539297 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.539299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.539826 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.539827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.539966 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.540561 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.540580 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.544033 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.546850 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.546851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.546852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.547092 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.547304 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.547305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.547306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.547369 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.547679 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.547680 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.547727 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.547789 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.549452 (+ 1663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":1692,"thread_start_us":57,"threads_started":1}
I20260923 15:11:51.550145 29678 heartbeater.cc:347] Connected to a master server at 127.21.204.125:39959
I20260923 15:11:51.550269 29678 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.550447 29678 heartbeater.cc:511] Master 127.21.204.125:39959 requested a full tablet report, sending...
I20260923 15:11:51.550127 29677 heartbeater.cc:347] Connected to a master server at 127.21.204.126:40489
I20260923 15:11:51.550661 29677 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.550730 29677 heartbeater.cc:511] Master 127.21.204.126:40489 requested a full tablet report, sending...
I20260923 15:11:51.550930 28431 ts_manager.cc:194] Re-registered known tserver with Master: 9d14836f0e77457ca7a893e002d211da (127.21.204.67:34017)
I20260923 15:11:51.550925 29029 ts_manager.cc:194] Registered new tserver with Master: 9d14836f0e77457ca7a893e002d211da (127.21.204.67:34017)
I20260923 15:11:51.551018 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.545573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.0.0.1:36196)
0923 15:11:51.545670 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.545679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.545720 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.547736 (+ 2016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.547737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.547739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:51.547775 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.547776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.548625 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.548626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.549798 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.549803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.550781 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.550782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.550849 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.550852 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.550900 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.550941 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 15:11:51.551280 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.545797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.21.204.124:43357)
0923 15:11:51.547579 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.547581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.547588 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:51.547607 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.547608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.547640 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.547697 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:51.547849 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.548530 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.550051 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.550614 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.550683 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.550748 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.551168 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.551211 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1769,"thread_start_us":66,"threads_started":1}
I20260923 15:11:51.551412 28431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:42949
I20260923 15:11:51.551478 29230 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 15:11:51.551573 29681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.534843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.21.204.67:41165)
0923 15:11:51.535033 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.535138 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.536285 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.536594 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.536599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.537036 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:51.537175 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.537177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.538174 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.538175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.539371 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.539374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.540048 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.540049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.540129 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.540632 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:51.540636 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:51.543785 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:51.545920 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.545927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.545929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.546247 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.546487 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:51.546488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:51.546488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:51.546619 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:51.546956 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:51.546963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:51.547124 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.551359 (+ 4235us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.551464 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":105,"thread_start_us":96,"threads_started":1}
I20260923 15:11:51.551617 29263 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03997f227c114b958667f054eaa101f6. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.551715 29263 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: LEARNER
I20260923 15:11:51.551926 29676 heartbeater.cc:347] Connected to a master server at 127.21.204.124:43357
I20260923 15:11:51.551972 29676 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:51.552122 29676 heartbeater.cc:511] Master 127.21.204.124:43357 requested a full tablet report, sending...
I20260923 15:11:51.552448 29218 ts_manager.cc:194] Registered new tserver with Master: 9d14836f0e77457ca7a893e002d211da (127.21.204.67:34017)
I20260923 15:11:51.553004 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.540449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.21.204.124:43357)
0923 15:11:51.542433 (+ 1984us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.542435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.542441 (+ 6us) server_negotiation.cc:407] Connection header received
0923 15:11:51.542461 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.542462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.542472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.542523 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.543212 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.545114 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.546146 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.546461 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.546524 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.548093 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.548094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.548095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.548101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.549650 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.550034 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.550035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.550035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.550089 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.552491 (+ 2402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.552492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.552492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.552630 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.552723 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.552824 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.552902 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1963,"thread_start_us":66,"threads_started":1}
I20260923 15:11:51.555492 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.539721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.21.204.126:40489)
0923 15:11:51.539868 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.539870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.540741 (+ 871us) server_negotiation.cc:407] Connection header received
0923 15:11:51.540867 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.540868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.540880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.540950 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.541759 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.542560 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.543612 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.543907 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.543948 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.552638 (+ 8690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.552640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.552640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.552648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.554420 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.554739 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.554740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.554741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.554801 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.554975 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.554976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.554977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.555093 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.555190 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.555309 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.555383 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260923 15:11:51.555969 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.540048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.21.204.125:39959)
0923 15:11:51.540816 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.540818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.541102 (+ 284us) server_negotiation.cc:407] Connection header received
0923 15:11:51.541124 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.541125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.541136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.541192 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.542403 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.543295 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.550486 (+ 7191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.550780 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.551265 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.552643 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.552644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.552644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.552650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.554871 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.555190 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.555192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.555192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.555245 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.555457 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.555458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.555458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.555599 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.555692 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.555793 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.555871 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":49,"threads_started":1}
I20260923 15:11:51.557077 28431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:51.558741 28431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:51.559542 29188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: NON_VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: true } } }
I20260923 15:11:51.559651 29188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.560460 29188 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: attempting to promote NON_VOTER d1c46dc772ee47fca057d6ee5c5586d7 to VOTER
I20260923 15:11:51.560595 29188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:51.561028 29230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 LEARNER]: Refusing update from remote peer 03997f227c114b958667f054eaa101f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 15:11:51.561069 29039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Refusing update from remote peer 03997f227c114b958667f054eaa101f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 15:11:51.561241 28990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 15:11:51.561340 28990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 15:11:51.562024 29184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.562300 29230 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.562934 29184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.563045 29184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.563126 29184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.563194 29184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.563247 29184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.563246 29263 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.563309 29184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:51.563319 29263 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:11:51.563418 29039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.564194 29183 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03997f227c114b958667f054eaa101f6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:51.564286 29183 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:11:51.569106 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.565132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:34017 (local address 127.0.0.1:59990)
0923 15:11:51.565265 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.565274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.565313 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.565699 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.565700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.565702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:51.565753 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.565755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.566873 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.566874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.567935 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.567937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.568873 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.568873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.568930 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.568936 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.568999 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.569036 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260923 15:11:51.569180 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.565343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.21.204.67:34017)
0923 15:11:51.565511 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.565515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.565526 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:11:51.565563 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.565566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.565583 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.565832 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:51.565928 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.566776 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.568022 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.568747 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.568868 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.568927 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.569015 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.569072 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260923 15:11:51.569550 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.565070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44703 (local address 127.0.0.1:51318)
0923 15:11:51.565111 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.565123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.565163 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.565662 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.565663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.565665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:51.565712 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.565713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.567189 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.567190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.568188 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.568191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.569346 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.569347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.569409 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.569413 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.569446 (+ 33us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.569481 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":1}
I20260923 15:11:51.569866 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.565662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:42863 (local address 127.0.0.1:54208)
0923 15:11:51.565788 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.565798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.565840 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.566264 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.566265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.566267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:51.566303 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.566304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.567560 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.567561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.568609 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.568611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.569555 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.569556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.569707 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.569711 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.569760 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.569795 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260923 15:11:51.570058 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.565213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.21.204.65:44703)
0923 15:11:51.565441 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.565446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.565464 (+ 18us) server_negotiation.cc:407] Connection header received
0923 15:11:51.565515 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.565517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.565531 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.565792 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:51.565855 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.567093 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.568314 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.569078 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.569739 (+ 661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.569819 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.569843 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.569901 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
I20260923 15:11:51.569912 29610 tablet_service.cc:1467] Processing CreateTablet for tablet 3c30181e2fad45f285ffe99e86b15b1b (DEFAULT_TABLE table=test-table [id=687e1bed032d432b9bff424ac87cc161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:51.569932 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.565896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.21.204.66:42863)
0923 15:11:51.566095 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.566099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.566109 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:51.566146 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.566148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.566163 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.566244 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:51.566366 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.567478 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.568675 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.569466 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.569539 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.569608 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.569775 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.569820 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260923 15:11:51.570314 29610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c30181e2fad45f285ffe99e86b15b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:51.570710 29337 tablet_service.cc:1467] Processing CreateTablet for tablet 3c30181e2fad45f285ffe99e86b15b1b (DEFAULT_TABLE table=test-table [id=687e1bed032d432b9bff424ac87cc161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:51.570674 29474 tablet_service.cc:1467] Processing CreateTablet for tablet 3c30181e2fad45f285ffe99e86b15b1b (DEFAULT_TABLE table=test-table [id=687e1bed032d432b9bff424ac87cc161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:51.571013 29474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c30181e2fad45f285ffe99e86b15b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:51.571020 29337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c30181e2fad45f285ffe99e86b15b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:51.574198 29706 tablet_bootstrap.cc:492] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Bootstrap starting.
I20260923 15:11:51.574832 29706 tablet_bootstrap.cc:654] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:51.575088 29706 log.cc:824] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:51.575204 29708 tablet_bootstrap.cc:492] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Bootstrap starting.
I20260923 15:11:51.575435 29707 tablet_bootstrap.cc:492] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Bootstrap starting.
I20260923 15:11:51.575848 29708 tablet_bootstrap.cc:654] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:51.576098 29707 tablet_bootstrap.cc:654] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:51.576781 29708 log.cc:824] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:51.576890 29707 log.cc:824] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:51.577765 29706 tablet_bootstrap.cc:492] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: No bootstrap required, opened a new log
I20260923 15:11:51.577765 29708 tablet_bootstrap.cc:492] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: No bootstrap required, opened a new log
I20260923 15:11:51.577837 29708 ts_tablet_manager.cc:1397] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:11:51.577837 29706 ts_tablet_manager.cc:1397] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 15:11:51.580989 29706 raft_consensus.cc:348] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.581194 29706 raft_consensus.cc:374] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:51.581235 29706 raft_consensus.cc:729] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d14836f0e77457ca7a893e002d211da, State: Initialized, Role: FOLLOWER
I20260923 15:11:51.581380 29706 consensus_queue.cc:260] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.581432 29708 raft_consensus.cc:348] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.581594 29708 raft_consensus.cc:374] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:51.581636 29708 raft_consensus.cc:729] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6dc5f13a1341dc9fc047f5f112bb74, State: Initialized, Role: FOLLOWER
I20260923 15:11:51.581662 29706 ts_tablet_manager.cc:1428] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260923 15:11:51.581755 29707 tablet_bootstrap.cc:492] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: No bootstrap required, opened a new log
I20260923 15:11:51.581780 29708 consensus_queue.cc:260] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.581837 29707 ts_tablet_manager.cc:1397] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 15:11:51.581853 29677 heartbeater.cc:503] Master 127.21.204.126:40489 was elected leader, sending a full tablet report...
I20260923 15:11:51.582049 29708 ts_tablet_manager.cc:1428] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260923 15:11:51.582790 29541 heartbeater.cc:503] Master 127.21.204.126:40489 was elected leader, sending a full tablet report...
I20260923 15:11:51.584172 29707 raft_consensus.cc:348] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.584378 29707 raft_consensus.cc:374] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:51.584425 29707 raft_consensus.cc:729] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce281b9d1e4434cb2a07ed3f186cb24, State: Initialized, Role: FOLLOWER
I20260923 15:11:51.584533 29707 consensus_queue.cc:260] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.584772 29707 ts_tablet_manager.cc:1428] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:11:51.584856 29404 heartbeater.cc:503] Master 127.21.204.126:40489 was elected leader, sending a full tablet report...
W20260923 15:11:51.651343 29543 tablet.cc:2369] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:51.681535 29713 raft_consensus.cc:482] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:51.681718 29713 raft_consensus.cc:504] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.682181 29713 leader_election.cc:288] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703), 9d14836f0e77457ca7a893e002d211da (127.21.204.67:34017)
I20260923 15:11:51.686116 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.682423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:44511 (local address 127.21.204.65:44703)
0923 15:11:51.682474 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.682476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.682487 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:11:51.682518 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.682519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.682529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.682596 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:51.682788 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.683552 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.685119 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.685711 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.685775 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.685783 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.686004 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.686047 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:11:51.686116 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.682423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:48613 (local address 127.21.204.67:34017)
0923 15:11:51.682480 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.682482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.682490 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:51.682518 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.682519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.682529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.682596 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:51.682792 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.683561 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.685097 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.685678 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.685766 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.685777 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.686007 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.686047 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 15:11:51.686131 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.682346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:34017 (local address 127.21.204.66:48613)
0923 15:11:51.682425 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.682434 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.682464 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.682670 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.682670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.682671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:51.682715 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.682716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.683674 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.683676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.684975 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.684979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.685808 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.685809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.685918 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.685923 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.685989 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.686039 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":89}
I20260923 15:11:51.686134 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.682338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44703 (local address 127.21.204.66:44511)
0923 15:11:51.682387 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:51.682399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:51.682443 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:51.682611 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:51.682612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:51.682614 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:51.682693 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.682696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.683688 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.683689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.684975 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:51.684979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.685808 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.685809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:51.685918 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.685923 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:11:51.685989 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 15:11:51.686039 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":61}
I20260923 15:11:51.686501 29630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c30181e2fad45f285ffe99e86b15b1b" candidate_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d14836f0e77457ca7a893e002d211da" is_pre_election: true
I20260923 15:11:51.686662 29630 raft_consensus.cc:2454] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b6dc5f13a1341dc9fc047f5f112bb74 in term 0.
I20260923 15:11:51.686879 29427 leader_election.cc:302] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b6dc5f13a1341dc9fc047f5f112bb74, 9d14836f0e77457ca7a893e002d211da; no voters:
I20260923 15:11:51.687006 29713 raft_consensus.cc:2781] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:51.687062 29713 raft_consensus.cc:482] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:51.687016 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c30181e2fad45f285ffe99e86b15b1b" candidate_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" is_pre_election: true
I20260923 15:11:51.687100 29713 raft_consensus.cc:3037] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:51.687162 29357 raft_consensus.cc:2454] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b6dc5f13a1341dc9fc047f5f112bb74 in term 0.
I20260923 15:11:51.687841 29713 raft_consensus.cc:504] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.687980 29713 leader_election.cc:288] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [CANDIDATE]: Term 1 election: Requested vote from peers 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703), 9d14836f0e77457ca7a893e002d211da (127.21.204.67:34017)
I20260923 15:11:51.688118 29630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c30181e2fad45f285ffe99e86b15b1b" candidate_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d14836f0e77457ca7a893e002d211da"
I20260923 15:11:51.688177 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c30181e2fad45f285ffe99e86b15b1b" candidate_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24"
I20260923 15:11:51.688222 29630 raft_consensus.cc:3037] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:51.688287 29357 raft_consensus.cc:3037] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:51.689050 29630 raft_consensus.cc:2454] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b6dc5f13a1341dc9fc047f5f112bb74 in term 1.
I20260923 15:11:51.689065 29357 raft_consensus.cc:2454] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b6dc5f13a1341dc9fc047f5f112bb74 in term 1.
I20260923 15:11:51.689224 29427 leader_election.cc:302] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b6dc5f13a1341dc9fc047f5f112bb74, 9d14836f0e77457ca7a893e002d211da; no voters:
I20260923 15:11:51.689352 29713 raft_consensus.cc:2781] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:51.689494 29713 raft_consensus.cc:686] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 LEADER]: Becoming Leader. State: Replica: 9b6dc5f13a1341dc9fc047f5f112bb74, State: Running, Role: LEADER
I20260923 15:11:51.689608 29713 consensus_queue.cc:237] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } }
I20260923 15:11:51.690657 28431 catalog_manager.cc:5495] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b6dc5f13a1341dc9fc047f5f112bb74 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } health_report { overall_health: HEALTHY } } }
I20260923 15:11:51.715931 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.712897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.21.204.66:42863)
0923 15:11:51.712974 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.712976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.712983 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:51.713016 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.713017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.713034 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.713123 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:51.713263 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.713781 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.714695 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.714894 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.714974 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.715240 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:51.715589 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:51.715738 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.715800 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.715821 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 15:11:51.718253 29630 raft_consensus.cc:1261] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 1 FOLLOWER]: Refusing update from remote peer 9b6dc5f13a1341dc9fc047f5f112bb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:51.718250 29357 raft_consensus.cc:1261] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Refusing update from remote peer 9b6dc5f13a1341dc9fc047f5f112bb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:51.718647 29713 consensus_queue.cc:1035] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:51.718778 29715 consensus_queue.cc:1035] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:51.723097 29720 mvcc.cc:205] Tried to move back new op lower bound from 7332562172794327040 to 7332562172681121792. Current Snapshot: MvccSnapshot[applied={T|T < 7332562172794327040}]
I20260923 15:11:51.724193 28430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:51.724295 28430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:51.725946 28430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03997f227c114b958667f054eaa101f6: Sending DeleteTablet for 3 replicas of tablet 3c30181e2fad45f285ffe99e86b15b1b
I20260923 15:11:51.726320 29474 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c30181e2fad45f285ffe99e86b15b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:51 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:54208
I20260923 15:11:51.726332 29610 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c30181e2fad45f285ffe99e86b15b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:51 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:59990
I20260923 15:11:51.726620 29337 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c30181e2fad45f285ffe99e86b15b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:51 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:51318
I20260923 15:11:51.726702 29728 tablet_replica.cc:330] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: stopping tablet replica
I20260923 15:11:51.726811 29728 raft_consensus.cc:2229] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:51.726920 29728 raft_consensus.cc:2258] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:51.727280 29727 tablet_replica.cc:330] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: stopping tablet replica
I20260923 15:11:51.727383 29727 raft_consensus.cc:2229] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:51.727596 29727 raft_consensus.cc:2258] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:51.727768 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 28399
I20260923 15:11:51.729075 29728 ts_tablet_manager.cc:1905] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:51.731067 29730 tablet_replica.cc:330] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: stopping tablet replica
I20260923 15:11:51.731333 29730 raft_consensus.cc:2229] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:51.731333 29728 ts_tablet_manager.cc:1918] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:51.731413 29728 log.cc:1196] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/3c30181e2fad45f285ffe99e86b15b1b
I20260923 15:11:51.731804 29728 ts_tablet_manager.cc:1939] T 3c30181e2fad45f285ffe99e86b15b1b P 9b6dc5f13a1341dc9fc047f5f112bb74: Deleting consensus metadata
I20260923 15:11:51.732127 29730 raft_consensus.cc:2258] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:51.732782 29730 ts_tablet_manager.cc:1905] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 15:11:51.735400 29427 connection.cc:535] server connection from 127.0.0.1:54208 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 15:11:51.736879 29727 ts_tablet_manager.cc:1905] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:51.738695 29727 ts_tablet_manager.cc:1918] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:51.738785 29727 log.cc:1196] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/3c30181e2fad45f285ffe99e86b15b1b
I20260923 15:11:51.739171 29727 ts_tablet_manager.cc:1939] T 3c30181e2fad45f285ffe99e86b15b1b P 9d14836f0e77457ca7a893e002d211da: Deleting consensus metadata
W20260923 15:11:51.739544 29563 connection.cc:441] server connection from 127.0.0.1:59990 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59990 (request call id 1) could send its response
I20260923 15:11:51.739640 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.726938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.21.204.124:43357)
0923 15:11:51.726989 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.726991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.726998 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:51.727020 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.727021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.727033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.727079 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.728675 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.729474 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.730728 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.731081 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.731235 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.735420 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.735422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.735422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.735431 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.737237 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.737993 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.737995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.737995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.738059 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.738918 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.738920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.738920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.739080 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.739177 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.739448 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.739544 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:11:51.740192 29730 ts_tablet_manager.cc:1918] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:51.740307 29730 log.cc:1196] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/3c30181e2fad45f285ffe99e86b15b1b
I20260923 15:11:51.740550 29730 ts_tablet_manager.cc:1939] T 3c30181e2fad45f285ffe99e86b15b1b P 4ce281b9d1e4434cb2a07ed3f186cb24: Deleting consensus metadata
W20260923 15:11:51.740949 29290 connection.cc:441] server connection from 127.0.0.1:51318 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51318 (request call id 1) could send its response
I20260923 15:11:51.747974 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.737367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.21.204.124:43357)
0923 15:11:51.737417 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.737419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.738130 (+ 711us) server_negotiation.cc:407] Connection header received
0923 15:11:51.738227 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.738227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.738250 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.738295 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.739021 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.739806 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.742199 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.742525 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.742564 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.744803 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.744804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.744805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.744812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.746594 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.747012 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.747013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.747013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.747076 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.747303 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.747304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.747304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.747440 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.747541 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.747705 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.747875 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:51.749178 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:51.737855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.21.204.125:39959)
0923 15:11:51.737927 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:51.737929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:51.742197 (+ 4268us) server_negotiation.cc:407] Connection header received
0923 15:11:51.742280 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:51.742282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:51.742301 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:51.742378 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:51.742970 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.743743 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.744512 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:51.744743 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:51.744831 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:51.746355 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:51.746356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:51.746357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:51.746380 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:51.748002 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.748413 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.748414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.748414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.748474 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:51.748678 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:51.748679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:51.748679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:51.748794 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:51.748896 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:51.748990 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 15:11:51.749079 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 15:11:52.724280 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:52.723929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:44011)
0923 15:11:52.724074 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:52.724207 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
W20260923 15:11:52.724677 29541 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:40489 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:52.724751 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:52.724190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:45137)
0923 15:11:52.724584 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:52.724701 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":72,"threads_started":1}
I20260923 15:11:52.724864 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:52.724490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:42805)
0923 15:11:52.724650 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:52.724719 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":45,"threads_started":1}
W20260923 15:11:52.724941 29404 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:40489 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
W20260923 15:11:52.725016 29677 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:40489 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:53.226348 29741 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03997f227c114b958667f054eaa101f6)
I20260923 15:11:53.226482 29741 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:53.226819 29741 leader_election.cc:288] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489), d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124:43357)
I20260923 15:11:53.226972 29742 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03997f227c114b958667f054eaa101f6)
I20260923 15:11:53.227100 29742 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:53.227196 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.226935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59558)
0923 15:11:53.227091 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.227143 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260923 15:11:53.227308 29014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:53.227360 29742 leader_election.cc:288] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489), eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125:39959)
I20260923 15:11:53.227787 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.227549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59564)
0923 15:11:53.227690 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.227741 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260923 15:11:53.227977 29209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:53.228247 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.227831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59570)
0923 15:11:53.228189 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.228218 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":60,"threads_started":1}
W20260923 15:11:53.228325 29014 leader_election.cc:334] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489): Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:53.228382 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.228290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59572)
0923 15:11:53.228327 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.228353 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260923 15:11:53.228441 29209 leader_election.cc:334] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489): Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:53.230562 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.226995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:43357 (local address 127.0.0.1:36214)
0923 15:11:53.227187 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.227305 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:53.227392 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:53.227678 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:53.227680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:53.227682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:53.227746 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:53.227748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.228856 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.228857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:53.229670 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:53.229672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.230299 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.230300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:53.230356 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:53.230360 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:53.230426 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 15:11:53.230468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":39,"threads_started":1}
I20260923 15:11:53.230619 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.227138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.21.204.124:43357)
0923 15:11:53.227232 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:53.227235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:53.227435 (+ 200us) server_negotiation.cc:407] Connection header received
0923 15:11:53.227535 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:53.227536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:53.227548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:53.227634 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:53.227995 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.228773 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.229758 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.230229 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.230283 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:53.230291 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:53.230505 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 15:11:53.230558 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":46,"threads_started":1}
I20260923 15:11:53.230803 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.227707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:39959 (local address 127.0.0.1:40420)
0923 15:11:53.227843 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.227853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:53.227900 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:53.228029 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:53.228029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:53.228031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:53.228077 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:53.228078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.229041 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.229042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:53.229928 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:53.229931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.230546 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.230547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:53.230613 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:53.230616 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:53.230681 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 15:11:53.230728 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":43,"threads_started":1}
I20260923 15:11:53.230881 29230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" is_pre_election: true
I20260923 15:11:53.230957 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.227791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.21.204.125:39959)
0923 15:11:53.227924 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:53.227926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:53.227935 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:53.227955 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:53.227956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:53.227965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:53.228027 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:53.228135 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.228958 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.230003 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:53.230434 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:53.230545 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:53.230553 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:53.230862 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 15:11:53.230907 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":86,"threads_started":1}
I20260923 15:11:53.230989 29230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb48151fe57d4268b99e218f1bcfd2ba in term 1.
I20260923 15:11:53.231165 29015 leader_election.cc:302] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1c46dc772ee47fca057d6ee5c5586d7, eb48151fe57d4268b99e218f1bcfd2ba; no voters: 03997f227c114b958667f054eaa101f6
I20260923 15:11:53.231170 29039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" is_pre_election: true
I20260923 15:11:53.231295 29741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 15:11:53.231366 29039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c46dc772ee47fca057d6ee5c5586d7 in term 1.
I20260923 15:11:53.231405 29741 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Starting leader election (detected failure of leader 03997f227c114b958667f054eaa101f6)
I20260923 15:11:53.231437 29741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 1 FOLLOWER]: Advancing to term 2
I20260923 15:11:53.231688 29209 leader_election.cc:302] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1c46dc772ee47fca057d6ee5c5586d7, eb48151fe57d4268b99e218f1bcfd2ba; no voters: 03997f227c114b958667f054eaa101f6
I20260923 15:11:53.231846 29742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 15:11:53.231896 29742 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 03997f227c114b958667f054eaa101f6)
I20260923 15:11:53.231916 29742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 1 FOLLOWER]: Advancing to term 2
I20260923 15:11:53.232407 29741 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:53.232555 29741 leader_election.cc:288] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [CANDIDATE]: Term 2 election: Requested vote from peers 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489), d1c46dc772ee47fca057d6ee5c5586d7 (127.21.204.124:43357)
I20260923 15:11:53.232774 29230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d1c46dc772ee47fca057d6ee5c5586d7"
I20260923 15:11:53.232818 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.232728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59580)
0923 15:11:53.232770 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.232794 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 15:11:53.232800 29742 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:53.232911 29230 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb48151fe57d4268b99e218f1bcfd2ba in current term 2: Already voted for candidate d1c46dc772ee47fca057d6ee5c5586d7 in this term.
I20260923 15:11:53.232929 29742 leader_election.cc:288] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 2 election: Requested vote from peers 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489), eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125:39959)
I20260923 15:11:53.233227 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.233135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59588)
0923 15:11:53.233173 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.233196 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260923 15:11:53.233328 29014 leader_election.cc:334] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489): Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:53.233387 29015 leader_election.cc:302] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb48151fe57d4268b99e218f1bcfd2ba; no voters: 03997f227c114b958667f054eaa101f6, d1c46dc772ee47fca057d6ee5c5586d7
I20260923 15:11:53.233408 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.233265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59598)
0923 15:11:53.233345 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.233377 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 15:11:53.233470 29741 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 15:11:53.233484 29039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
I20260923 15:11:53.233564 29039 raft_consensus.cc:2379] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d1c46dc772ee47fca057d6ee5c5586d7 in current term 2: Already voted for candidate eb48151fe57d4268b99e218f1bcfd2ba in this term.
I20260923 15:11:53.233776 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.233691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59610)
0923 15:11:53.233725 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.233751 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260923 15:11:53.233836 29209 leader_election.cc:334] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489): Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:53.233865 29209 leader_election.cc:302] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c46dc772ee47fca057d6ee5c5586d7; no voters: 03997f227c114b958667f054eaa101f6, eb48151fe57d4268b99e218f1bcfd2ba
I20260923 15:11:53.233961 29742 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 15:11:53.725332 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.725058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:39795)
0923 15:11:53.725222 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.725261 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260923 15:11:53.725493 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.725276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:40073)
0923 15:11:53.725415 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.725452 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260923 15:11:53.725490 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:53.725196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:51805)
0923 15:11:53.725405 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:53.725443 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260923 15:11:54.726063 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:54.725723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:36463)
0923 15:11:54.725876 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:54.726000 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260923 15:11:54.726225 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:54.725901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:55381)
0923 15:11:54.726133 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:54.726177 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":47,"threads_started":1}
I20260923 15:11:54.726253 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:54.725968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:54613)
0923 15:11:54.726181 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:54.726210 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":59,"threads_started":1}
I20260923 15:11:55.073173 29758 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:55.073273 29758 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:55.073408 29758 leader_election.cc:288] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489), eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125:39959)
I20260923 15:11:55.073818 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.073600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59882)
0923 15:11:55.073717 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.073765 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260923 15:11:55.073933 29039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" is_pre_election: true
I20260923 15:11:55.074038 29039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c46dc772ee47fca057d6ee5c5586d7 in term 2.
I20260923 15:11:55.074317 29209 leader_election.cc:302] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1c46dc772ee47fca057d6ee5c5586d7, eb48151fe57d4268b99e218f1bcfd2ba; no voters:
I20260923 15:11:55.074460 29758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 15:11:55.074530 29758 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:55.074568 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.074274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59912)
0923 15:11:55.074365 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.074456 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 15:11:55.074568 29758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 2 FOLLOWER]: Advancing to term 3
W20260923 15:11:55.074654 29209 leader_election.cc:334] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489): Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:55.075336 29758 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:55.075464 29758 leader_election.cc:288] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 3 election: Requested vote from peers 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489), eb48151fe57d4268b99e218f1bcfd2ba (127.21.204.125:39959)
I20260923 15:11:55.075696 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.075589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59914)
0923 15:11:55.075647 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.075678 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 15:11:55.075774 29039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb48151fe57d4268b99e218f1bcfd2ba"
I20260923 15:11:55.075865 29039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 2 FOLLOWER]: Advancing to term 3
I20260923 15:11:55.076004 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.075909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59922)
0923 15:11:55.075951 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.075976 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 15:11:55.076108 29209 leader_election.cc:334] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489): Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
I20260923 15:11:55.076601 29039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c46dc772ee47fca057d6ee5c5586d7 in term 3.
I20260923 15:11:55.076761 29209 leader_election.cc:302] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1c46dc772ee47fca057d6ee5c5586d7, eb48151fe57d4268b99e218f1bcfd2ba; no voters: 03997f227c114b958667f054eaa101f6
I20260923 15:11:55.076884 29758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 15:11:55.076992 29758 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [term 3 LEADER]: Becoming Leader. State: Replica: d1c46dc772ee47fca057d6ee5c5586d7, State: Running, Role: LEADER
I20260923 15:11:55.077080 29758 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } }
I20260923 15:11:55.077423 29761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c46dc772ee47fca057d6ee5c5586d7. Latest consensus state: current_term: 3 leader_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:55.077521 29761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:55.077696 29763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:55.078189 29763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:11:55.078251 29763 catalog_manager.cc:1259] Loaded cluster ID: d1f82c282da24552925d540452d15f68
I20260923 15:11:55.078269 29763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:55.078408 29763 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:55.078457 29763 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Loaded TSK: 0
I20260923 15:11:55.078550 29763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:11:55.159387 29221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:55.160930 29221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:55.162057 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.161909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59944)
0923 15:11:55.161952 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.162018 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 15:11:55.162156 29039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [term 3 FOLLOWER]: Refusing update from remote peer d1c46dc772ee47fca057d6ee5c5586d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 15:11:55.162391 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.162292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.0.0.1:59948)
0923 15:11:55.162323 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.162368 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260923 15:11:55.162389 29761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260923 15:11:55.162503 29209 consensus_peers.cc:487] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 -> Peer 03997f227c114b958667f054eaa101f6 (127.21.204.126:40489): Couldn't send request to peer 03997f227c114b958667f054eaa101f6. Status: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 15:11:55.163043 29765 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c46dc772ee47fca057d6ee5c5586d7. Latest consensus state: current_term: 3 leader_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:55.163154 29765 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:11:55.163677 29766 mvcc.cc:205] Tried to move back new op lower bound from 7332562186901307392 to 7332562186556313600. Current Snapshot: MvccSnapshot[applied={T|T < 7332562186901307392}]
I20260923 15:11:55.163653 29758 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:55.163746 29758 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7 [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:55.163853 29765 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "03997f227c114b958667f054eaa101f6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 40489 } } peers { permanent_uuid: "eb48151fe57d4268b99e218f1bcfd2ba" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 39959 } attrs { promote: false } } peers { permanent_uuid: "d1c46dc772ee47fca057d6ee5c5586d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 43357 } attrs { promote: false } } }
I20260923 15:11:55.163941 29765 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb48151fe57d4268b99e218f1bcfd2ba [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:11:55.169364 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.165682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:34017 (local address 127.0.0.1:59992)
0923 15:11:55.165840 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.165850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:55.165992 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:55.166252 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:55.166252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:55.166265 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:55.166332 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.166334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.167085 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.167086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.168317 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.168320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.169128 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.169129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.169184 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.169187 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:55.169233 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 15:11:55.169269 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":19}
I20260923 15:11:55.169473 29771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.165717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44703 (local address 127.0.0.1:51334)
0923 15:11:55.166088 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.166098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:55.166135 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:55.166301 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:55.166302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:55.166305 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:55.166358 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.166361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.167290 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.167291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.168445 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.168446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.169196 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.169197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.169259 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.169263 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:55.169353 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 15:11:55.169396 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260923 15:11:55.169473 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.165998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.21.204.65:44703)
0923 15:11:55.166129 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:55.166131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:55.166140 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:55.166196 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:55.166197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:55.166205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:55.166281 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:55.166442 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.167222 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.168531 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.169104 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.169190 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.169201 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:55.169362 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 15:11:55.169408 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260923 15:11:55.169697 29337 tablet_service.cc:1467] Processing CreateTablet for tablet 00c0c59ad31e4ef996d270fbf8e83d1f (DEFAULT_TABLE table=test-table [id=800b61eac96d472892520208ac7165f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:55.169742 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.165741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.21.204.67:34017)
0923 15:11:55.165914 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:55.165917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:55.165926 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:55.166138 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:55.166139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:55.166151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:55.166418 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:55.166456 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.167013 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.168429 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.169023 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.169126 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.169140 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:55.169329 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 15:11:55.169381 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
I20260923 15:11:55.169855 29337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c0c59ad31e4ef996d270fbf8e83d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:55.169932 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.165759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.21.204.66:42863)
0923 15:11:55.166248 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:55.166250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:55.166346 (+ 96us) server_negotiation.cc:407] Connection header received
0923 15:11:55.166416 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:55.166417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:55.166430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:55.166488 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:55.166585 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.167310 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.168557 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.168943 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.169026 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.169038 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:55.169684 (+ 646us) server_negotiation.cc:299] Negotiation successful
0923 15:11:55.169745 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":56,"threads_started":1}
I20260923 15:11:55.170078 29610 tablet_service.cc:1467] Processing CreateTablet for tablet 00c0c59ad31e4ef996d270fbf8e83d1f (DEFAULT_TABLE table=test-table [id=800b61eac96d472892520208ac7165f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:55.169282 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.165683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:42863 (local address 127.0.0.1:54230)
0923 15:11:55.166292 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.166301 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:55.166351 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:55.166497 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:55.166498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:55.166500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:55.166532 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.166533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.167369 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.167370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.168461 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.168464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.169024 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.169025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.169103 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.169108 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:11:55.169158 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 15:11:55.169201 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":59,"thread_start_us":73,"threads_started":1}
I20260923 15:11:55.170181 29610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c0c59ad31e4ef996d270fbf8e83d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:55.170398 29474 tablet_service.cc:1467] Processing CreateTablet for tablet 00c0c59ad31e4ef996d270fbf8e83d1f (DEFAULT_TABLE table=test-table [id=800b61eac96d472892520208ac7165f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:55.170519 29474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00c0c59ad31e4ef996d270fbf8e83d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:55.171535 29776 tablet_bootstrap.cc:492] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: Bootstrap starting.
I20260923 15:11:55.171614 29777 tablet_bootstrap.cc:492] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Bootstrap starting.
I20260923 15:11:55.171945 29777 tablet_bootstrap.cc:654] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:55.171942 29775 tablet_bootstrap.cc:492] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24: Bootstrap starting.
I20260923 15:11:55.171974 29776 tablet_bootstrap.cc:654] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:55.172369 29775 tablet_bootstrap.cc:654] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:55.172525 29777 tablet_bootstrap.cc:492] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: No bootstrap required, opened a new log
I20260923 15:11:55.172530 29776 tablet_bootstrap.cc:492] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: No bootstrap required, opened a new log
I20260923 15:11:55.172574 29776 ts_tablet_manager.cc:1397] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:55.172585 29777 ts_tablet_manager.cc:1397] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:55.172739 29777 raft_consensus.cc:348] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.172710 29776 raft_consensus.cc:348] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.172811 29776 raft_consensus.cc:374] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:55.172837 29776 raft_consensus.cc:729] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6dc5f13a1341dc9fc047f5f112bb74, State: Initialized, Role: FOLLOWER
I20260923 15:11:55.172888 29775 tablet_bootstrap.cc:492] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24: No bootstrap required, opened a new log
I20260923 15:11:55.172879 29776 consensus_queue.cc:260] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.172925 29775 ts_tablet_manager.cc:1397] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:55.172822 29777 raft_consensus.cc:374] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:55.172977 29777 raft_consensus.cc:729] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d14836f0e77457ca7a893e002d211da, State: Initialized, Role: FOLLOWER
I20260923 15:11:55.173045 29776 ts_tablet_manager.cc:1428] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:55.173043 29777 consensus_queue.cc:260] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.173044 29775 raft_consensus.cc:348] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.173120 29775 raft_consensus.cc:374] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:55.173161 29775 raft_consensus.cc:729] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce281b9d1e4434cb2a07ed3f186cb24, State: Initialized, Role: FOLLOWER
I20260923 15:11:55.173202 29775 consensus_queue.cc:260] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.173252 29777 ts_tablet_manager.cc:1428] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:55.173422 29775 ts_tablet_manager.cc:1428] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:55.173448 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.173347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:51347)
0923 15:11:55.173387 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.173420 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260923 15:11:55.173691 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.173532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:53463)
0923 15:11:55.173634 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.173664 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 15:11:55.173792 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.173697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:43231)
0923 15:11:55.173734 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.173767 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260923 15:11:55.174288 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.174163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:49585)
0923 15:11:55.174242 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.174265 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 15:11:55.174394 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.174283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:58549)
0923 15:11:55.174346 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.174373 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 15:11:55.174567 29540 heartbeater.cc:503] Master 127.21.204.124:43357 was elected leader, sending a full tablet report...
I20260923 15:11:55.174793 29676 heartbeater.cc:503] Master 127.21.204.124:43357 was elected leader, sending a full tablet report...
W20260923 15:11:55.258746 29406 tablet.cc:2369] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:55.287026 29679 tablet.cc:2369] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:55.362968 29782 raft_consensus.cc:482] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:55.363062 29782 raft_consensus.cc:504] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.363454 29782 leader_election.cc:288] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b6dc5f13a1341dc9fc047f5f112bb74 (127.21.204.66:42863), 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703)
I20260923 15:11:55.366353 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.363732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:51285 (local address 127.21.204.65:44703)
0923 15:11:55.363784 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:55.363787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:55.363838 (+ 51us) server_negotiation.cc:407] Connection header received
0923 15:11:55.363919 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:55.363920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:55.363934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:55.363994 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:55.364155 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.364635 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.365624 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.366008 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.366098 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.366109 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:55.366251 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 15:11:55.366285 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:11:55.366402 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.363760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:33963 (local address 127.21.204.66:42863)
0923 15:11:55.363807 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:55.363809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:55.363928 (+ 119us) server_negotiation.cc:407] Connection header received
0923 15:11:55.364004 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:55.364005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:55.364019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:55.364077 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:55.364229 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.364694 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.365627 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.366055 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.366134 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.366145 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:55.366293 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 15:11:55.366329 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:11:55.366412 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.363658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44703 (local address 127.21.204.67:51285)
0923 15:11:55.363753 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.363768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:55.363847 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:55.363997 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:55.363998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:55.364000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:55.364085 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.364087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.364736 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.364738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.365529 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.365531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.366099 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.366100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.366174 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.366177 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:55.366245 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 15:11:55.366283 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":26}
I20260923 15:11:55.366526 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.363676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:42863 (local address 127.21.204.67:33963)
0923 15:11:55.363853 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.363862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:55.363936 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:55.364079 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:55.364080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:55.364082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:55.364160 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.364161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.364775 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.364776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.365535 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:55.365537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.366154 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.366155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:55.366220 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.366223 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:55.366286 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 15:11:55.366327 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260923 15:11:55.366746 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0c59ad31e4ef996d270fbf8e83d1f" candidate_uuid: "9d14836f0e77457ca7a893e002d211da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" is_pre_election: true
I20260923 15:11:55.366783 29494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0c59ad31e4ef996d270fbf8e83d1f" candidate_uuid: "9d14836f0e77457ca7a893e002d211da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" is_pre_election: true
I20260923 15:11:55.366865 29357 raft_consensus.cc:2454] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d14836f0e77457ca7a893e002d211da in term 0.
I20260923 15:11:55.366899 29494 raft_consensus.cc:2454] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d14836f0e77457ca7a893e002d211da in term 0.
I20260923 15:11:55.367059 29566 leader_election.cc:302] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce281b9d1e4434cb2a07ed3f186cb24, 9d14836f0e77457ca7a893e002d211da; no voters:
I20260923 15:11:55.367174 29782 raft_consensus.cc:2781] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:55.367218 29782 raft_consensus.cc:482] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:55.367238 29782 raft_consensus.cc:3037] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:55.368008 29782 raft_consensus.cc:504] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.368276 29494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0c59ad31e4ef996d270fbf8e83d1f" candidate_uuid: "9d14836f0e77457ca7a893e002d211da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74"
I20260923 15:11:55.368288 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00c0c59ad31e4ef996d270fbf8e83d1f" candidate_uuid: "9d14836f0e77457ca7a893e002d211da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24"
I20260923 15:11:55.368366 29494 raft_consensus.cc:3037] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:55.368397 29357 raft_consensus.cc:3037] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:55.368996 29494 raft_consensus.cc:2454] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d14836f0e77457ca7a893e002d211da in term 1.
I20260923 15:11:55.368214 29782 leader_election.cc:288] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [CANDIDATE]: Term 1 election: Requested vote from peers 9b6dc5f13a1341dc9fc047f5f112bb74 (127.21.204.66:42863), 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703)
I20260923 15:11:55.369130 29357 raft_consensus.cc:2454] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d14836f0e77457ca7a893e002d211da in term 1.
I20260923 15:11:55.369220 29565 leader_election.cc:302] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b6dc5f13a1341dc9fc047f5f112bb74, 9d14836f0e77457ca7a893e002d211da; no voters:
I20260923 15:11:55.369319 29782 raft_consensus.cc:2781] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:55.369479 29782 raft_consensus.cc:686] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 1 LEADER]: Becoming Leader. State: Replica: 9d14836f0e77457ca7a893e002d211da, State: Running, Role: LEADER
I20260923 15:11:55.369556 29782 consensus_queue.cc:237] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } }
I20260923 15:11:55.369912 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.369790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:35799)
0923 15:11:55.369826 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.369864 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 15:11:55.370302 29218 catalog_manager.cc:5495] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d14836f0e77457ca7a893e002d211da (127.21.204.67). New cstate: current_term: 1 leader_uuid: "9d14836f0e77457ca7a893e002d211da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d14836f0e77457ca7a893e002d211da" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:55.396334 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.394001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.21.204.67:34017)
0923 15:11:55.394070 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:55.394072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:55.394081 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:11:55.394117 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:55.394119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:55.394142 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:55.394226 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:55.394353 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.394833 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.395573 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:55.395776 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:55.395847 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:55.396003 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:55.396109 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:55.396188 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:55.396229 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 15:11:55.396249 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:11:55.397578 29357 raft_consensus.cc:1261] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Refusing update from remote peer 9d14836f0e77457ca7a893e002d211da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:55.397640 29494 raft_consensus.cc:1261] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 FOLLOWER]: Refusing update from remote peer 9d14836f0e77457ca7a893e002d211da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:55.397923 29782 consensus_queue.cc:1035] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce281b9d1e4434cb2a07ed3f186cb24" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:55.398098 29785 consensus_queue.cc:1035] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b6dc5f13a1341dc9fc047f5f112bb74" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 42863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:55.398958 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.398843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:53607)
0923 15:11:55.398890 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.398925 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 15:11:55.399153 29403 heartbeater.cc:503] Master 127.21.204.124:43357 was elected leader, sending a full tablet report...
I20260923 15:11:55.399760 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.399643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:33605)
0923 15:11:55.399690 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.399727 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 15:11:55.400035 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.399913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:38437)
0923 15:11:55.399981 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.400007 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 15:11:55.400740 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.400616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.66:35705)
0923 15:11:55.400680 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.400715 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 15:11:55.401294 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.401197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.67:45925)
0923 15:11:55.401234 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.401267 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260923 15:11:55.401482 29791 mvcc.cc:205] Tried to move back new op lower bound from 7332562187865767936 to 7332562187754291200. Current Snapshot: MvccSnapshot[applied={T|T < 7332562187865767936}]
I20260923 15:11:55.401624 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:55.401508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:40489 (local address 127.21.204.65:54767)
0923 15:11:55.401570 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.401596 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 15:11:55.402058 29789 mvcc.cc:205] Tried to move back new op lower bound from 7332562187865767936 to 7332562187754291200. Current Snapshot: MvccSnapshot[applied={T|T < 7332562187865767936 or (T in {7332562187865767936})}]
I20260923 15:11:55.404170 29218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:55.404268 29218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:55.405578 29218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1c46dc772ee47fca057d6ee5c5586d7: Sending DeleteTablet for 3 replicas of tablet 00c0c59ad31e4ef996d270fbf8e83d1f
I20260923 15:11:55.405848 29474 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0c59ad31e4ef996d270fbf8e83d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:55 UTC) from {username='oryx', principal='oryx/127.21.204.124@KRBTEST.COM'} at 127.0.0.1:54230
I20260923 15:11:55.405944 29337 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0c59ad31e4ef996d270fbf8e83d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:55 UTC) from {username='oryx', principal='oryx/127.21.204.124@KRBTEST.COM'} at 127.0.0.1:51334
I20260923 15:11:55.405990 29610 tablet_service.cc:1514] Processing DeleteTablet for tablet 00c0c59ad31e4ef996d270fbf8e83d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:55 UTC) from {username='oryx', principal='oryx/127.21.204.124@KRBTEST.COM'} at 127.0.0.1:59992
I20260923 15:11:55.406152 29799 tablet_replica.cc:330] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24: stopping tablet replica
I20260923 15:11:55.406204 29799 raft_consensus.cc:2229] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:55.406251 29799 raft_consensus.cc:2258] T 00c0c59ad31e4ef996d270fbf8e83d1f P 4ce281b9d1e4434cb2a07ed3f186cb24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:55.406503 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 29177
I20260923 15:11:55.408246 29800 tablet_replica.cc:330] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: stopping tablet replica
I20260923 15:11:55.408321 29800 raft_consensus.cc:2229] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:55.408416 29800 raft_consensus.cc:2258] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:55.408622 29798 tablet_replica.cc:330] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: stopping tablet replica
I20260923 15:11:55.408679 29798 raft_consensus.cc:2229] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:55.408730 29798 raft_consensus.cc:2258] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:55.408924 29800 ts_tablet_manager.cc:1905] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:55.410636 29800 ts_tablet_manager.cc:1918] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:55.410745 29800 log.cc:1196] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/00c0c59ad31e4ef996d270fbf8e83d1f
I20260923 15:11:55.410811 29798 ts_tablet_manager.cc:1905] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:55.411121 29800 ts_tablet_manager.cc:1939] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9d14836f0e77457ca7a893e002d211da: Deleting consensus metadata
I20260923 15:11:55.411602 29208 catalog_manager.cc:4841] TS 9d14836f0e77457ca7a893e002d211da (127.21.204.67:34017): tablet 00c0c59ad31e4ef996d270fbf8e83d1f (table test-table [id=800b61eac96d472892520208ac7165f3]) successfully deleted
W20260923 15:11:55.412197 29773 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:55.412059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44703 (local address 127.0.0.1:51340)
0923 15:11:55.412105 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.412129 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44703: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 15:11:55.412413 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 29411
W20260923 15:11:55.412436 29211 catalog_manager.cc:4584] TS 4ce281b9d1e4434cb2a07ed3f186cb24 (127.21.204.65:44703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00c0c59ad31e4ef996d270fbf8e83d1f: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44703: connect: Connection reset by peer (error 104)
I20260923 15:11:55.412484 29798 ts_tablet_manager.cc:1918] T 00c0c59ad31e4ef996d270fbf8e83d1f P 9b6dc5f13a1341dc9fc047f5f112bb74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260923 15:11:55.417943 29773 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:55.417818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:42863 (local address 127.0.0.1:54234)
0923 15:11:55.417862 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:55.417906 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.66:42863: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 15:11:55.418149 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 29547
I20260923 15:11:55.423990 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 28895
I20260923 15:11:55.429278 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 29083
2026-09-23T15:11:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6785 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:55 dist-test-slave-t5lt krb5kdc[29803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:55 dist-test-slave-t5lt krb5kdc[29803](info): set up 2 sockets
Sep 23 15:11:55 dist-test-slave-t5lt krb5kdc[29803](info): commencing operation
krb5kdc: starting...
W20260923 15:11:56.484129 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:56 dist-test-slave-t5lt krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab.
Entry for principal oryx/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab.
I20260923 15:11:56.605526 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42975
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab
--principal=oryx/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:42975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:56.686270 29816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:56.686496 29816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:56.686532 29816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:56.687697 29816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:56.687759 29816 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 15:11:56.687781 29816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:56.687804 29816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:56.687830 29816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:56.687853 29816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:56.689380 29816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:42975
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.126.keytab
--principal=oryx/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:56.689656 29816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:56.689895 29816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:56.694600 29822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:56.694729 29824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:56.694605 29821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:56.694854 29816 server_base.cc:1034] running on GCE node
I20260923 15:11:56.695123 29816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:56.695410 29816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:56.696599 29816 hybrid_clock.cc:648] HybridClock initialized: now 1790176316696567 us; error 59 us; skew 500 ppm
Sep 23 15:11:56 dist-test-slave-t5lt krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176316, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:56.699151 29816 init.cc:377] Logged in from keytab as oryx/127.21.204.126@KRBTEST.COM (short username oryx)
I20260923 15:11:56.699584 29816 webserver.cc:460] Webserver started at http://127.21.204.126:45337/ using document root <none> and password file <none>
I20260923 15:11:56.699813 29816 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:56.699898 29816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:56.700037 29816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:56.701098 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "3b2052df1714479ea9d588719e6c408c"
format_stamp: "Formatted at 2026-09-23 15:11:56 on dist-test-slave-t5lt"
server_key: "f653142e08eb49161d70adce9f588909"
server_key_iv: "46f985e788b30e7cbf3759f592d5acb0"
server_key_version: "encryptionkey@0"
I20260923 15:11:56.701464 29816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "3b2052df1714479ea9d588719e6c408c"
format_stamp: "Formatted at 2026-09-23 15:11:56 on dist-test-slave-t5lt"
server_key: "f653142e08eb49161d70adce9f588909"
server_key_iv: "46f985e788b30e7cbf3759f592d5acb0"
server_key_version: "encryptionkey@0"
I20260923 15:11:56.702904 29816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:11:56.703639 29831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:56.703845 29816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:56.703948 29816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "3b2052df1714479ea9d588719e6c408c"
format_stamp: "Formatted at 2026-09-23 15:11:56 on dist-test-slave-t5lt"
server_key: "f653142e08eb49161d70adce9f588909"
server_key_iv: "46f985e788b30e7cbf3759f592d5acb0"
server_key_version: "encryptionkey@0"
I20260923 15:11:56.704020 29816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:56.718878 29816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:56.720773 29816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:56.720940 29816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:56.725474 29816 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:42975
I20260923 15:11:56.725564 29893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:42975 every 8 connection(s)
I20260923 15:11:56.725896 29816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:11:56.726596 29894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:56.728873 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Bootstrap starting.
I20260923 15:11:56.729501 29894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:56.729966 29894 log.cc:824] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:56.730360 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 29816
I20260923 15:11:56.730460 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:11:56.730685 22321 external_mini_cluster.cc:1428] Setting key dc793e0422c1633c375a87e4b572a323
I20260923 15:11:56.730870 29894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: No bootstrap required, opened a new log
I20260923 15:11:56.733686 29894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2052df1714479ea9d588719e6c408c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42975 } }
I20260923 15:11:56.733842 29894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:56.733911 29894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2052df1714479ea9d588719e6c408c, State: Initialized, Role: FOLLOWER
I20260923 15:11:56.734040 29894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2052df1714479ea9d588719e6c408c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42975 } }
I20260923 15:11:56.734122 29894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:11:56.734165 29894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:11:56.734221 29894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:56.734882 29894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2052df1714479ea9d588719e6c408c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42975 } }
I20260923 15:11:56.734993 29894 leader_election.cc:302] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b2052df1714479ea9d588719e6c408c; no voters:
I20260923 15:11:56.735141 29894 leader_election.cc:288] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:11:56.735185 29899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:56.735338 29899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [term 1 LEADER]: Becoming Leader. State: Replica: 3b2052df1714479ea9d588719e6c408c, State: Running, Role: LEADER
I20260923 15:11:56.735384 29894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:11:56.735442 29899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2052df1714479ea9d588719e6c408c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42975 } }
I20260923 15:11:56.735743 29899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2052df1714479ea9d588719e6c408c. Latest consensus state: current_term: 1 leader_uuid: "3b2052df1714479ea9d588719e6c408c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2052df1714479ea9d588719e6c408c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42975 } } }
I20260923 15:11:56.735862 29899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [sys.catalog]: This master's current role is: LEADER
I20260923 15:11:56.735756 29901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b2052df1714479ea9d588719e6c408c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b2052df1714479ea9d588719e6c408c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42975 } } }
I20260923 15:11:56.736002 29901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c [sys.catalog]: This master's current role is: LEADER
W20260923 15:11:56.737319 29914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:11:56.737383 29914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:11:56.737525 29905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:11:56.737757 29905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 15:11:56 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:56.739554 29905 catalog_manager.cc:1347] Generated new cluster ID: 53d351ba44cc4889b2b0d95bf025abe7
I20260923 15:11:56.739605 29905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:11:56.743422 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:56.731226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.21.204.126:42975)
0923 15:11:56.731393 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:56.731406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:56.731431 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:11:56.732180 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:56.732188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:56.732237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:56.732348 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:56.733280 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.734394 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.735481 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.736181 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.736265 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:56.739253 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:56.739256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:56.739257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:56.739277 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:56.741319 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:56.741697 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:56.741700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:56.741700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:56.741810 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:56.742038 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:56.742039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:56.742039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:56.742350 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:56.742980 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:56.743060 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 15:11:56.743200 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260923 15:11:56.768096 29905 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:11:56.768885 29905 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:11:56.775314 29905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Generated new TSK 0
I20260923 15:11:56.775488 29905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab.
Entry for principal oryx/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab.
I20260923 15:11:56.813227 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42975
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab
--principal=oryx/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:56.894199 29920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:56.894392 29920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:56.894415 29920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:56.895498 29920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:56.895534 29920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:56.895555 29920 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:56.895572 29920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:11:56.897006 29920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab
--principal=oryx/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:42975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:56.897186 29920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:56.897346 29920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:11:56.902549 29920 server_base.cc:1034] running on GCE node
W20260923 15:11:56.902503 29925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:56.902514 29928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:56.902540 29926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:56.903028 29920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:56.903213 29920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:56.904349 29920 hybrid_clock.cc:648] HybridClock initialized: now 1790176316904340 us; error 26 us; skew 500 ppm
Sep 23 15:11:56 dist-test-slave-t5lt krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176316, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:56.906914 29920 init.cc:377] Logged in from keytab as oryx/127.21.204.65@KRBTEST.COM (short username oryx)
I20260923 15:11:56.907332 29920 webserver.cc:460] Webserver started at http://127.21.204.65:43015/ using document root <none> and password file <none>
I20260923 15:11:56.907562 29920 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:56.907621 29920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:56.907817 29920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:56.908825 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "b3b4b34b37434d9e978796452bcc2e95"
format_stamp: "Formatted at 2026-09-23 15:11:56 on dist-test-slave-t5lt"
server_key: "cd4626d9ddae90d965d7cd9cfeb90d74"
server_key_iv: "ae99d6afe19778a226e408679571d905"
server_key_version: "encryptionkey@0"
I20260923 15:11:56.909168 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "b3b4b34b37434d9e978796452bcc2e95"
format_stamp: "Formatted at 2026-09-23 15:11:56 on dist-test-slave-t5lt"
server_key: "cd4626d9ddae90d965d7cd9cfeb90d74"
server_key_iv: "ae99d6afe19778a226e408679571d905"
server_key_version: "encryptionkey@0"
I20260923 15:11:56.910595 29920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:56.911355 29935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:56.911533 29920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:56.911634 29920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "b3b4b34b37434d9e978796452bcc2e95"
format_stamp: "Formatted at 2026-09-23 15:11:56 on dist-test-slave-t5lt"
server_key: "cd4626d9ddae90d965d7cd9cfeb90d74"
server_key_iv: "ae99d6afe19778a226e408679571d905"
server_key_version: "encryptionkey@0"
I20260923 15:11:56.911705 29920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:56.937034 29920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:56.939194 29920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:56.939368 29920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:56.939919 29920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:56.939985 29920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:56.940034 29920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:56.940079 29920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:56.947170 29920 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:46701
I20260923 15:11:56.947201 30050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:46701 every 8 connection(s)
Sep 23 15:11:56 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176316, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:56.947597 29920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:11:56.948194 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 29920
I20260923 15:11:56.948325 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:11:56.948572 22321 external_mini_cluster.cc:1428] Setting key e76c0cf3f784baf34ffde7b6d493275e
I20260923 15:11:56.951249 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:56.940468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:59609 (local address 127.21.204.126:42975)
0923 15:11:56.940516 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:56.940519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:56.940766 (+ 247us) server_negotiation.cc:407] Connection header received
0923 15:11:56.941259 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:56.941261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:56.941276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:56.941365 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:56.942235 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.943189 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.944759 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.945180 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.945271 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:56.947965 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:56.947967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:56.947967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:56.947975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:56.949759 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:56.950238 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:56.950239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:56.950239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:56.950306 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:56.950545 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:56.950546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:56.950546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:56.950692 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:56.950765 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:56.951110 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 15:11:56.951185 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:11:56.951601 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:56.940472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42975 (local address 127.21.204.65:59609)
0923 15:11:56.940633 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:56.940667 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:56.941152 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:56.941444 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:56.941449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:56.941803 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:56.942116 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:56.942124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.943326 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.943328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:56.944646 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:56.944649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.945277 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.945279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:56.945406 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:56.945928 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:56.945940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:56.947861 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:56.949837 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:56.949844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:56.949845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:56.950164 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:56.950374 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:56.950375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:56.950375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:56.950476 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:56.950767 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:56.950776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:56.950958 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 15:11:56.951077 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 15:11:56.951322 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260923 15:11:56.953220 30051 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42975
I20260923 15:11:56.953308 30051 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:56.953491 30051 heartbeater.cc:511] Master 127.21.204.126:42975 requested a full tablet report, sending...
I20260923 15:11:56.954088 29847 ts_manager.cc:194] Registered new tserver with Master: b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701)
I20260923 15:11:56.955188 29847 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:59609
WARNING: no policy specified for oryx/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:11:56.959690 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:56.953365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:51173 (local address 127.21.204.126:42975)
0923 15:11:56.953417 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:56.953420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:56.954028 (+ 608us) server_negotiation.cc:407] Connection header received
0923 15:11:56.954101 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:56.954102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:56.954112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:56.954194 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:56.954517 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.954983 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.955868 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.956072 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.956151 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:56.957052 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:56.957054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:56.957054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:56.957060 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:56.958553 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:56.958891 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:56.958893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:56.958893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:56.958946 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:56.959171 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:56.959172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:56.959172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:56.959281 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:56.959349 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:56.959521 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 15:11:56.959593 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 15:11:56.959699 30058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:56.953309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42975 (local address 127.21.204.65:51173)
0923 15:11:56.953969 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:56.953978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:56.954042 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:56.954231 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:56.954232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:56.954388 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:56.954442 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:56.954443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.955074 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.955075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:56.955788 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:56.955790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:56.956154 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:56.956155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:56.956225 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:56.956650 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:56.956655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:56.956952 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:56.958643 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:56.958644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:56.958645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:56.958811 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:56.959040 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:56.959041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:56.959041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:56.959100 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:56.959355 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:56.959356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:56.959410 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 15:11:56.959494 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 15:11:56.959593 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab.
Entry for principal oryx/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab.
I20260923 15:11:56.976608 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42975
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab
--principal=oryx/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:57.058279 30060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:57.058485 30060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:57.058509 30060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:57.059651 30060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:57.059700 30060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:57.059720 30060 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:57.059758 30060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:11:57.061252 30060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab
--principal=oryx/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:42975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:57.061488 30060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:57.061717 30060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:57.066596 30068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:57.066684 30066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:57.066606 30065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:57.066794 30060 server_base.cc:1034] running on GCE node
I20260923 15:11:57.067000 30060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:57.067256 30060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:57.068414 30060 hybrid_clock.cc:648] HybridClock initialized: now 1790176317068393 us; error 29 us; skew 500 ppm
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:57.070844 30060 init.cc:377] Logged in from keytab as oryx/127.21.204.66@KRBTEST.COM (short username oryx)
I20260923 15:11:57.071313 30060 webserver.cc:460] Webserver started at http://127.21.204.66:35329/ using document root <none> and password file <none>
I20260923 15:11:57.071542 30060 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:57.071609 30060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:57.071753 30060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:57.072751 30060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "a6a70824b4844d96a775a5a8dc323476"
format_stamp: "Formatted at 2026-09-23 15:11:57 on dist-test-slave-t5lt"
server_key: "c9cf93146f3e72934af11d61f0cc30b5"
server_key_iv: "0d380c16e12c440273e58cafabb401ea"
server_key_version: "encryptionkey@0"
I20260923 15:11:57.073096 30060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "a6a70824b4844d96a775a5a8dc323476"
format_stamp: "Formatted at 2026-09-23 15:11:57 on dist-test-slave-t5lt"
server_key: "c9cf93146f3e72934af11d61f0cc30b5"
server_key_iv: "0d380c16e12c440273e58cafabb401ea"
server_key_version: "encryptionkey@0"
I20260923 15:11:57.074488 30060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:57.075189 30075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.075345 30060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:57.075444 30060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "a6a70824b4844d96a775a5a8dc323476"
format_stamp: "Formatted at 2026-09-23 15:11:57 on dist-test-slave-t5lt"
server_key: "c9cf93146f3e72934af11d61f0cc30b5"
server_key_iv: "0d380c16e12c440273e58cafabb401ea"
server_key_version: "encryptionkey@0"
I20260923 15:11:57.075516 30060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:57.087920 30060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:57.089787 30060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:57.089957 30060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:57.090509 30060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:57.090567 30060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.090606 30060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:57.090626 30060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:57.097836 30060 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:35825
I20260923 15:11:57.098112 30190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:35825 every 8 connection(s)
I20260923 15:11:57.098265 30060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:11:57.100492 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.090927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:39589 (local address 127.21.204.126:42975)
0923 15:11:57.090993 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.090996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.091163 (+ 167us) server_negotiation.cc:407] Connection header received
0923 15:11:57.091735 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.091737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.091751 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.091837 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.092461 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.093307 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.094549 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.094916 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.094999 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.097495 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:57.097497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:57.097497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:57.097506 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:57.099106 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.099546 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.099548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.099548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.099605 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.099856 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.099857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.099858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.099975 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:57.100049 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.100322 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.100400 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:11:57.100674 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.090909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42975 (local address 127.21.204.66:39589)
0923 15:11:57.091051 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.091091 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.091634 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.091929 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.091932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.092207 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:57.092385 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.092390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.093404 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.093405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.094482 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.094484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.095018 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.095019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.095165 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.095634 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:57.095650 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:57.097397 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:57.099198 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.099206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.099208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.099456 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.099673 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.099674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.099675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.099790 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.100038 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:57.100045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:57.100194 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.100300 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.100454 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260923 15:11:57.101094 30191 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42975
I20260923 15:11:57.101358 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 30060
I20260923 15:11:57.101481 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:11:57.101713 22321 external_mini_cluster.cc:1428] Setting key e3e5b93e451458b960db374bdae61a9f
I20260923 15:11:57.101867 30191 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:57.102082 30191 heartbeater.cc:511] Master 127.21.204.126:42975 requested a full tablet report, sending...
I20260923 15:11:57.102423 29848 ts_manager.cc:194] Registered new tserver with Master: a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825)
I20260923 15:11:57.102926 29848 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:39589
I20260923 15:11:57.108390 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.102097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42975 (local address 127.21.204.66:58079)
0923 15:11:57.102366 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.102378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.102431 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.102545 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.102546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.102710 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:57.102768 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.102770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.103529 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.103530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.104309 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.104312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.104646 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.104647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.104716 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.105197 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:57.105202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:57.105489 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:57.107313 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.107315 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.107315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.107486 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.107711 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.107711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.107712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.107768 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.108130 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:57.108131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:57.108183 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.108241 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.108320 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260923 15:11:57.108482 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.102125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:58079 (local address 127.21.204.126:42975)
0923 15:11:57.102159 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.102160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.102419 (+ 259us) server_negotiation.cc:407] Connection header received
0923 15:11:57.102476 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.102477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.102487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.102540 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.102843 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.103439 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.104397 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.104573 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.104631 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.105570 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:57.105571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:57.105571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:57.105579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:57.107233 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.107571 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.107572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.107572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.107634 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.107833 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.107834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.107834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.108047 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:57.108127 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.108293 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.108376 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
WARNING: no policy specified for oryx/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab.
Entry for principal oryx/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab.
I20260923 15:11:57.129190 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42975
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab
--principal=oryx/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:57.210525 30200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:57.210772 30200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:57.210803 30200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:57.211897 30200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:57.211931 30200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:57.211951 30200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 15:11:57.211968 30200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:11:57.213366 30200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab
--principal=oryx/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.204.126:42975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:57.213598 30200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:57.213828 30200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:57.218853 30205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:57.218864 30206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:57.218945 30200 server_base.cc:1034] running on GCE node
W20260923 15:11:57.218861 30208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:57.219260 30200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:57.219518 30200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:57.220700 30200 hybrid_clock.cc:648] HybridClock initialized: now 1790176317220690 us; error 28 us; skew 500 ppm
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:11:57.223215 30200 init.cc:377] Logged in from keytab as oryx/127.21.204.67@KRBTEST.COM (short username oryx)
I20260923 15:11:57.223657 30200 webserver.cc:460] Webserver started at http://127.21.204.67:46165/ using document root <none> and password file <none>
I20260923 15:11:57.223832 30200 fs_manager.cc:359] Metadata directory not provided
I20260923 15:11:57.223870 30200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:11:57.223958 30200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:11:57.224912 30200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "1c8adf0e64c5461391675fea399f02ba"
format_stamp: "Formatted at 2026-09-23 15:11:57 on dist-test-slave-t5lt"
server_key: "f638b01a007db706230fa1d41d8cbe21"
server_key_iv: "01b991451ddbea23b0a0a3598916c77b"
server_key_version: "encryptionkey@0"
I20260923 15:11:57.225198 30200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "1c8adf0e64c5461391675fea399f02ba"
format_stamp: "Formatted at 2026-09-23 15:11:57 on dist-test-slave-t5lt"
server_key: "f638b01a007db706230fa1d41d8cbe21"
server_key_iv: "01b991451ddbea23b0a0a3598916c77b"
server_key_version: "encryptionkey@0"
I20260923 15:11:57.226604 30200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:11:57.227347 30215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.227511 30200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:11:57.227598 30200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "1c8adf0e64c5461391675fea399f02ba"
format_stamp: "Formatted at 2026-09-23 15:11:57 on dist-test-slave-t5lt"
server_key: "f638b01a007db706230fa1d41d8cbe21"
server_key_iv: "01b991451ddbea23b0a0a3598916c77b"
server_key_version: "encryptionkey@0"
I20260923 15:11:57.227654 30200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:11:57.244702 30200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:11:57.246937 30200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:11:57.247049 30200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:11:57.247493 30200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:11:57.247525 30200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.247550 30200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:11:57.247606 30200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:57.254693 30200 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:43295
I20260923 15:11:57.254751 30330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:43295 every 8 connection(s)
I20260923 15:11:57.255045 30200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:11:57.258062 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.248046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:45305 (local address 127.21.204.126:42975)
0923 15:11:57.248118 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.248120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.248342 (+ 222us) server_negotiation.cc:407] Connection header received
0923 15:11:57.248837 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.248839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.248854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.248940 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.249788 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.250665 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.252126 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.252443 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.252536 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.255229 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:57.255230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:57.255231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:57.255239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:57.256675 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.257095 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.257096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.257096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.257154 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.257402 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.257403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.257403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.257521 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:57.257595 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.257857 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.257966 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:11:57.258271 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.248025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42975 (local address 127.21.204.67:45305)
0923 15:11:57.248197 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.248237 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.248727 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.248993 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.248996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.249376 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:57.249676 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.249683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.250772 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.250774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.252030 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.252033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.252570 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.252571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.252763 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.253279 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:57.253290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:57.255124 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:57.256784 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.256791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.256793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.257010 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.257217 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.257218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.257219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.257336 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.257585 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:57.257592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:57.257730 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.257837 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.258010 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260923 15:11:57.258731 30331 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42975
I20260923 15:11:57.258829 30331 heartbeater.cc:464] Registering TS with master...
I20260923 15:11:57.259002 30331 heartbeater.cc:511] Master 127.21.204.126:42975 requested a full tablet report, sending...
I20260923 15:11:57.259395 29847 ts_manager.cc:194] Registered new tserver with Master: 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.259927 29847 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:45305
I20260923 15:11:57.264178 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 30200
I20260923 15:11:57.264314 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:11:57.264503 22321 external_mini_cluster.cc:1428] Setting key dc129a302a579d2c09258bfe37a6940b
I20260923 15:11:57.265062 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.259791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:44081 (local address 127.21.204.126:42975)
0923 15:11:57.259839 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.259841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.259901 (+ 60us) server_negotiation.cc:407] Connection header received
0923 15:11:57.259963 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.259964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.259971 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.260054 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.260405 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.260865 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.261553 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.261760 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.261829 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.262664 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:57.262666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:57.262666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:57.262673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:57.263857 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.264268 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.264270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.264270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.264337 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.264535 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.264536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.264537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.264662 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:57.264737 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.264901 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.264973 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:11:57.265076 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.259722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42975 (local address 127.21.204.67:44081)
0923 15:11:57.259847 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.259857 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.259909 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.260062 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.260063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.260251 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:57.260320 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.260321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.260952 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.260953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.261471 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.261473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.261828 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.261829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.261929 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.262320 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:57.262324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:57.262578 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:57.263937 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.263939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.263939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.264150 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.264405 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.264406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.264407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.264470 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.264752 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:57.264753 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:57.264799 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.264872 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.264976 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260923 15:11:57.265239 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 15:11:57.286504 29898 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:57.280805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.21.204.126:42975)
0923 15:11:57.280864 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.280866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.280874 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:57.280898 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.280899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.280907 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.280965 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.281514 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.282070 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.282869 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.283055 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.283133 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.286383 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48242: BlockingRecv error: recv got EOF from 127.0.0.1:48242 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:57.293638 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.287204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.21.204.126:42975)
0923 15:11:57.287257 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.287259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.287303 (+ 44us) server_negotiation.cc:407] Connection header received
0923 15:11:57.287357 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.287357 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.287365 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.287413 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.287855 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.288302 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.288953 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.289127 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.289174 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.291236 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:57.291237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:57.291238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:57.291243 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:57.292669 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.293011 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.293012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.293013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.293087 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.293254 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.293255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.293255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.293353 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:57.293427 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.293503 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.293561 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 15:11:57.295281 29847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:57.296854 29847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:57.303275 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.299425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:43295 (local address 127.0.0.1:43208)
0923 15:11:57.299789 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.299799 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.299865 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.300143 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.300144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.300146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:57.300195 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.300196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.301370 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.301371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.302290 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.302292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.302974 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.302976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.303053 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.303067 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.303139 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.303192 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260923 15:11:57.303318 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.299645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.21.204.67:43295)
0923 15:11:57.299984 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.299988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.299998 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:57.300032 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.300034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.300048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.300264 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:57.300323 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.301287 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.302372 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.302873 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.302969 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.303048 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.303180 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.303218 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":81,"threads_started":1}
I20260923 15:11:57.303594 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.299198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35825 (local address 127.0.0.1:57218)
0923 15:11:57.299391 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.299434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.299500 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.299713 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.299716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.299718 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:57.299789 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.299791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.301189 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.301190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.302340 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.302341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.303290 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.303291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.303367 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.303370 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.303415 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.303464 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":11,"thread_start_us":82,"threads_started":1}
I20260923 15:11:57.303650 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.299302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.21.204.66:35825)
0923 15:11:57.299482 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.299488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.299500 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:11:57.299596 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.299599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.299615 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.299706 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:57.299867 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.301100 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.302423 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.303100 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.303192 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.303262 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.303506 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.303555 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260923 15:11:57.303735 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.299330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.21.204.65:46701)
0923 15:11:57.299655 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.299660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.299670 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:57.299707 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.299709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.299724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.299796 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:57.300005 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.300848 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.302417 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.303117 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.303179 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.303247 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.303571 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.303615 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":70,"threads_started":1}
I20260923 15:11:57.303844 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.299226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:46701 (local address 127.0.0.1:55528)
0923 15:11:57.299491 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.299505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.299551 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.299864 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.299864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.299866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:57.299907 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.299908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.300929 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.300930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.302313 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.302316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.303286 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.303288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.303362 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.303365 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.303415 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.303464 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260923 15:11:57.304351 30125 tablet_service.cc:1467] Processing CreateTablet for tablet b1f019a40d284639a79ec9d463e7c407 (DEFAULT_TABLE table=test-table [id=b0cb57d582d34c579584285af2bf5cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.304479 29985 tablet_service.cc:1467] Processing CreateTablet for tablet b1f019a40d284639a79ec9d463e7c407 (DEFAULT_TABLE table=test-table [id=b0cb57d582d34c579584285af2bf5cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.304479 30265 tablet_service.cc:1467] Processing CreateTablet for tablet b1f019a40d284639a79ec9d463e7c407 (DEFAULT_TABLE table=test-table [id=b0cb57d582d34c579584285af2bf5cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.304744 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f019a40d284639a79ec9d463e7c407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.304811 30265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f019a40d284639a79ec9d463e7c407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.304811 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1f019a40d284639a79ec9d463e7c407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.308032 30359 tablet_bootstrap.cc:492] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Bootstrap starting.
I20260923 15:11:57.308655 30360 tablet_bootstrap.cc:492] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Bootstrap starting.
I20260923 15:11:57.308696 30359 tablet_bootstrap.cc:654] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.308904 30361 tablet_bootstrap.cc:492] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Bootstrap starting.
I20260923 15:11:57.309142 30359 log.cc:824] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:57.309370 30360 tablet_bootstrap.cc:654] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.309535 30361 tablet_bootstrap.cc:654] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.309700 30360 log.cc:824] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:57.309849 30361 log.cc:824] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Log is configured to *not* fsync() on all Append() calls
I20260923 15:11:57.310032 30359 tablet_bootstrap.cc:492] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: No bootstrap required, opened a new log
I20260923 15:11:57.310087 30359 ts_tablet_manager.cc:1397] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:11:57.310532 30361 tablet_bootstrap.cc:492] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: No bootstrap required, opened a new log
I20260923 15:11:57.310585 30361 ts_tablet_manager.cc:1397] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:11:57.311717 30359 raft_consensus.cc:348] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.311869 30359 raft_consensus.cc:374] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.311914 30359 raft_consensus.cc:729] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a70824b4844d96a775a5a8dc323476, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.312031 30359 consensus_queue.cc:260] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.312260 30361 raft_consensus.cc:348] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.312345 30191 heartbeater.cc:503] Master 127.21.204.126:42975 was elected leader, sending a full tablet report...
I20260923 15:11:57.312379 30361 raft_consensus.cc:374] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.312418 30361 raft_consensus.cc:729] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8adf0e64c5461391675fea399f02ba, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.312527 30361 consensus_queue.cc:260] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.312604 30359 ts_tablet_manager.cc:1428] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:11:57.312772 30361 ts_tablet_manager.cc:1428] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:11:57.312911 30331 heartbeater.cc:503] Master 127.21.204.126:42975 was elected leader, sending a full tablet report...
I20260923 15:11:57.313374 30360 tablet_bootstrap.cc:492] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: No bootstrap required, opened a new log
I20260923 15:11:57.313446 30360 ts_tablet_manager.cc:1397] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260923 15:11:57.315269 30360 raft_consensus.cc:348] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.315408 30360 raft_consensus.cc:374] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.315454 30360 raft_consensus.cc:729] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.315562 30360 consensus_queue.cc:260] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.315814 30360 ts_tablet_manager.cc:1428] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 15:11:57.315877 30051 heartbeater.cc:503] Master 127.21.204.126:42975 was elected leader, sending a full tablet report...
W20260923 15:11:57.349520 30192 tablet.cc:2369] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:11:57.448431 30052 tablet.cc:2369] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:57.471170 30367 raft_consensus.cc:482] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:57.471288 30367 raft_consensus.cc:504] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.471621 30367 leader_election.cc:288] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.475140 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.472444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:60437 (local address 127.21.204.67:43295)
0923 15:11:57.472489 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.472491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.472499 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:57.472526 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.472526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.472535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.472648 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:57.472951 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.473503 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.474428 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.474789 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.474872 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.474881 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.475021 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.475059 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:11:57.475154 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.472218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:50077 (local address 127.21.204.66:35825)
0923 15:11:57.472265 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.472267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.472336 (+ 69us) server_negotiation.cc:407] Connection header received
0923 15:11:57.472393 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.472394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.472406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.472485 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:57.472723 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.473405 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.474416 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.474780 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.474874 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.474882 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.475041 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.475080 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:11:57.475186 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.472050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35825 (local address 127.21.204.65:50077)
0923 15:11:57.472204 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.472216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.472273 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.472534 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.472534 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.472537 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:57.472626 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.472629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.473486 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.473486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.474332 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.474335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.474879 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.474880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.474969 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.474972 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.475036 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.475084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":26,"thread_start_us":94,"threads_started":1}
I20260923 15:11:57.475194 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.472282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:43295 (local address 127.21.204.65:60437)
0923 15:11:57.472404 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.472414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.472458 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.472677 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.472678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.472681 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:57.472797 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.472800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.473593 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.473594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.474342 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.474344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.474871 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.474871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.474949 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.474953 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.475014 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.475055 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":1,"thread_start_us":42,"threads_started":1}
I20260923 15:11:57.475534 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f019a40d284639a79ec9d463e7c407" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba" is_pre_election: true
I20260923 15:11:57.475512 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f019a40d284639a79ec9d463e7c407" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a70824b4844d96a775a5a8dc323476" is_pre_election: true
I20260923 15:11:57.475672 30285 raft_consensus.cc:2454] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 0.
I20260923 15:11:57.475672 30145 raft_consensus.cc:2454] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 0.
I20260923 15:11:57.475893 29936 leader_election.cc:302] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8adf0e64c5461391675fea399f02ba, b3b4b34b37434d9e978796452bcc2e95; no voters:
I20260923 15:11:57.476032 30367 raft_consensus.cc:2781] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:57.476101 30367 raft_consensus.cc:482] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:57.476125 30367 raft_consensus.cc:3037] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.476824 30367 raft_consensus.cc:504] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.476948 30367 leader_election.cc:288] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 election: Requested vote from peers a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.477130 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f019a40d284639a79ec9d463e7c407" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a70824b4844d96a775a5a8dc323476"
I20260923 15:11:57.477164 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1f019a40d284639a79ec9d463e7c407" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba"
I20260923 15:11:57.477232 30285 raft_consensus.cc:3037] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.477226 30145 raft_consensus.cc:3037] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.478214 30285 raft_consensus.cc:2454] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 1.
I20260923 15:11:57.478214 30145 raft_consensus.cc:2454] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 1.
I20260923 15:11:57.478394 29936 leader_election.cc:302] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a70824b4844d96a775a5a8dc323476, b3b4b34b37434d9e978796452bcc2e95; no voters:
I20260923 15:11:57.478483 30367 raft_consensus.cc:2781] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:57.478629 30367 raft_consensus.cc:686] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 1 LEADER]: Becoming Leader. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Running, Role: LEADER
I20260923 15:11:57.478878 30367 consensus_queue.cc:237] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.479693 29848 catalog_manager.cc:5495] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65). New cstate: current_term: 1 leader_uuid: "b3b4b34b37434d9e978796452bcc2e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:57.487000 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.484825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.21.204.65:46701)
0923 15:11:57.484876 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.484877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.484885 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:11:57.484917 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.484918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.484933 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.484995 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:57.485106 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.485538 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.486261 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.486505 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.486560 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.486688 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:57.486779 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:57.486841 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.486887 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.486909 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 15:11:57.488771 30285 raft_consensus.cc:1261] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Refusing update from remote peer b3b4b34b37434d9e978796452bcc2e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.488772 30145 raft_consensus.cc:1261] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Refusing update from remote peer b3b4b34b37434d9e978796452bcc2e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.489114 30370 consensus_queue.cc:1035] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.489238 30367 consensus_queue.cc:1035] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.491448 30373 mvcc.cc:205] Tried to move back new op lower bound from 7332562196431495168 to 7332562196394102784. Current Snapshot: MvccSnapshot[applied={T|T < 7332562196431495168}]
I20260923 15:11:57.492569 30375 mvcc.cc:205] Tried to move back new op lower bound from 7332562196431495168 to 7332562196394102784. Current Snapshot: MvccSnapshot[applied={T|T < 7332562196431495168}]
I20260923 15:11:57.494048 29847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:57.494150 29847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:57.494845 29847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Sending DeleteTablet for 3 replicas of tablet b1f019a40d284639a79ec9d463e7c407
I20260923 15:11:57.495193 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet b1f019a40d284639a79ec9d463e7c407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:55528
I20260923 15:11:57.495193 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet b1f019a40d284639a79ec9d463e7c407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:57218
I20260923 15:11:57.495194 30265 tablet_service.cc:1514] Processing DeleteTablet for tablet b1f019a40d284639a79ec9d463e7c407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:43208
I20260923 15:11:57.495426 30384 tablet_replica.cc:330] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: stopping tablet replica
I20260923 15:11:57.495523 30384 raft_consensus.cc:2229] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:57.495640 29847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:57.495779 29847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:57.496294 30384 raft_consensus.cc:2258] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.496918 29985 tablet_service.cc:1467] Processing CreateTablet for tablet b05a71b3c05a490f9ccdf62f2e993fdc (DEFAULT_TABLE table=test-table [id=321f4ceaae1b4685b7ea9e8577223751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.496951 30265 tablet_service.cc:1467] Processing CreateTablet for tablet b05a71b3c05a490f9ccdf62f2e993fdc (DEFAULT_TABLE table=test-table [id=321f4ceaae1b4685b7ea9e8577223751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.497042 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05a71b3c05a490f9ccdf62f2e993fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.497044 30265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05a71b3c05a490f9ccdf62f2e993fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.497100 30384 ts_tablet_manager.cc:1905] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.498489 30360 tablet_bootstrap.cc:492] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: Bootstrap starting.
I20260923 15:11:57.498548 30361 tablet_bootstrap.cc:492] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Bootstrap starting.
I20260923 15:11:57.498694 30384 ts_tablet_manager.cc:1918] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:57.498757 30384 log.cc:1196] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/b1f019a40d284639a79ec9d463e7c407
I20260923 15:11:57.498919 30360 tablet_bootstrap.cc:654] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.498963 30361 tablet_bootstrap.cc:654] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.499015 30384 ts_tablet_manager.cc:1939] T b1f019a40d284639a79ec9d463e7c407 P a6a70824b4844d96a775a5a8dc323476: Deleting consensus metadata
I20260923 15:11:57.499366 30125 tablet_service.cc:1467] Processing CreateTablet for tablet b05a71b3c05a490f9ccdf62f2e993fdc (DEFAULT_TABLE table=test-table [id=321f4ceaae1b4685b7ea9e8577223751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.499454 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b05a71b3c05a490f9ccdf62f2e993fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.499447 30383 tablet_replica.cc:330] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: stopping tablet replica
I20260923 15:11:57.499538 30383 raft_consensus.cc:2229] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:57.499634 30383 raft_consensus.cc:2258] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.500324 30359 tablet_bootstrap.cc:492] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: Bootstrap starting.
I20260923 15:11:57.500682 29832 catalog_manager.cc:4841] TS a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825): tablet b1f019a40d284639a79ec9d463e7c407 (table test-table [id=b0cb57d582d34c579584285af2bf5cb6]) successfully deleted
I20260923 15:11:57.500732 30359 tablet_bootstrap.cc:654] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.501329 30383 ts_tablet_manager.cc:1905] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.501478 30361 tablet_bootstrap.cc:492] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: No bootstrap required, opened a new log
I20260923 15:11:57.501529 30361 ts_tablet_manager.cc:1397] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 15:11:57.501683 30361 raft_consensus.cc:348] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.501765 30361 raft_consensus.cc:374] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.501792 30361 raft_consensus.cc:729] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8adf0e64c5461391675fea399f02ba, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.501833 30361 consensus_queue.cc:260] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.501959 30361 ts_tablet_manager.cc:1428] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 15:11:57.502959 30383 ts_tablet_manager.cc:1918] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:57.503037 30383 log.cc:1196] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/b1f019a40d284639a79ec9d463e7c407
I20260923 15:11:57.503297 30383 ts_tablet_manager.cc:1939] T b1f019a40d284639a79ec9d463e7c407 P 1c8adf0e64c5461391675fea399f02ba: Deleting consensus metadata
I20260923 15:11:57.503640 30359 tablet_bootstrap.cc:492] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: No bootstrap required, opened a new log
I20260923 15:11:57.503696 30359 ts_tablet_manager.cc:1397] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 15:11:57.503823 29832 catalog_manager.cc:4841] TS 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295): tablet b1f019a40d284639a79ec9d463e7c407 (table test-table [id=b0cb57d582d34c579584285af2bf5cb6]) successfully deleted
I20260923 15:11:57.503849 30359 raft_consensus.cc:348] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.503916 30359 raft_consensus.cc:374] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.503942 30359 raft_consensus.cc:729] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a70824b4844d96a775a5a8dc323476, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.503988 30359 consensus_queue.cc:260] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.504076 30359 ts_tablet_manager.cc:1428] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260923 15:11:57.505518 30332 tablet.cc:2369] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:11:57.505993 30385 tablet_replica.cc:330] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: stopping tablet replica
I20260923 15:11:57.506099 30385 raft_consensus.cc:2229] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:57.506215 30385 raft_consensus.cc:2258] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.506299 30360 tablet_bootstrap.cc:492] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: No bootstrap required, opened a new log
I20260923 15:11:57.506352 30360 ts_tablet_manager.cc:1397] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.000s
I20260923 15:11:57.506493 30360 raft_consensus.cc:348] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.506574 30360 raft_consensus.cc:374] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.506603 30360 raft_consensus.cc:729] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.506633 30385 ts_tablet_manager.cc:1905] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.506647 30360 consensus_queue.cc:260] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.506793 30360 ts_tablet_manager.cc:1428] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.508296 30385 ts_tablet_manager.cc:1918] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:57.508368 30385 log.cc:1196] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/b1f019a40d284639a79ec9d463e7c407
I20260923 15:11:57.508603 30385 ts_tablet_manager.cc:1939] T b1f019a40d284639a79ec9d463e7c407 P b3b4b34b37434d9e978796452bcc2e95: Deleting consensus metadata
I20260923 15:11:57.508965 29834 catalog_manager.cc:4841] TS b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701): tablet b1f019a40d284639a79ec9d463e7c407 (table test-table [id=b0cb57d582d34c579584285af2bf5cb6]) successfully deleted
I20260923 15:11:57.572439 30365 raft_consensus.cc:482] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:57.572563 30365 raft_consensus.cc:504] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.572947 30365 leader_election.cc:288] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.576730 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.573330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:50897 (local address 127.21.204.67:43295)
0923 15:11:57.573377 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.573379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.573390 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:11:57.573420 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.573421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.573429 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.573489 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:57.573632 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.574238 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.575825 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.576361 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.576448 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.576458 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.576609 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.576650 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:11:57.576763 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.573132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:43295 (local address 127.21.204.66:50897)
0923 15:11:57.573307 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.573317 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.573368 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.573491 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.573492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.573493 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:57.573568 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.573570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.574317 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.574318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.575722 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.575725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.576466 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.576467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.576519 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.576523 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.576587 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.576662 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":67,"thread_start_us":57,"threads_started":1}
I20260923 15:11:57.577172 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05a71b3c05a490f9ccdf62f2e993fdc" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba" is_pre_election: true
I20260923 15:11:57.577281 30285 raft_consensus.cc:2454] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 0.
I20260923 15:11:57.577410 30084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.573111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:46701 (local address 127.21.204.66:42543)
0923 15:11:57.573168 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.573181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.573223 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.573441 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.573442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.573443 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:11:57.573509 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.573511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.574406 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.574406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.575722 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.575725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.577136 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.577137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.577224 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.577227 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.577286 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.577328 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":4}
I20260923 15:11:57.577450 30076 leader_election.cc:302] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8adf0e64c5461391675fea399f02ba, a6a70824b4844d96a775a5a8dc323476; no voters:
I20260923 15:11:57.577520 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.573231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:42543 (local address 127.21.204.65:46701)
0923 15:11:57.573279 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.573281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.573291 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:11:57.573321 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.573322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.573335 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.573390 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:11:57.573575 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.574320 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.576494 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.577045 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.577129 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.577137 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.577418 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.577457 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:11:57.577592 30365 raft_consensus.cc:2781] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:57.577636 30365 raft_consensus.cc:482] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:57.577661 30365 raft_consensus.cc:3037] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.577824 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05a71b3c05a490f9ccdf62f2e993fdc" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b4b34b37434d9e978796452bcc2e95" is_pre_election: true
I20260923 15:11:57.577983 30005 raft_consensus.cc:2454] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 0.
I20260923 15:11:57.578519 30365 raft_consensus.cc:504] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.578686 30365 leader_election.cc:288] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 election: Requested vote from peers b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.578825 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05a71b3c05a490f9ccdf62f2e993fdc" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba"
I20260923 15:11:57.578895 30285 raft_consensus.cc:3037] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.579435 30285 raft_consensus.cc:2454] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 1.
I20260923 15:11:57.579550 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b05a71b3c05a490f9ccdf62f2e993fdc" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b4b34b37434d9e978796452bcc2e95"
I20260923 15:11:57.579593 30076 leader_election.cc:302] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8adf0e64c5461391675fea399f02ba, a6a70824b4844d96a775a5a8dc323476; no voters:
I20260923 15:11:57.579620 30005 raft_consensus.cc:3037] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.579684 30365 raft_consensus.cc:2781] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:57.579790 30365 raft_consensus.cc:686] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 1 LEADER]: Becoming Leader. State: Replica: a6a70824b4844d96a775a5a8dc323476, State: Running, Role: LEADER
I20260923 15:11:57.579859 30365 consensus_queue.cc:237] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.580400 30005 raft_consensus.cc:2454] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 1.
I20260923 15:11:57.580531 29847 catalog_manager.cc:5495] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a70824b4844d96a775a5a8dc323476 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } health_report { overall_health: UNKNOWN } } }
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176316, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.126@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:57.614221 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.606765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.21.204.126:42975)
0923 15:11:57.606831 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.606834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.607003 (+ 169us) server_negotiation.cc:407] Connection header received
0923 15:11:57.607073 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.607074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.607084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.607142 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.607762 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.608261 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.609137 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.609342 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.609397 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.611832 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:57.611833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:57.611833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:57.611841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:57.613027 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.613397 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.613398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.613399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.613452 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.613652 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.613653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.613653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.613766 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:57.613835 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.614003 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.614086 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:11:57.614321 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.606722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42975 (local address 127.0.0.1:48264)
0923 15:11:57.606897 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.606910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.607001 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.607167 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.607168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.607550 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:11:57.607607 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.607610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.608339 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.608340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.609056 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.609059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.609416 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.609417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.609475 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.610096 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:11:57.610101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:11:57.611746 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:11:57.613116 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.613123 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.613126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.613312 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.613514 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:11:57.613515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:11:57.613516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:11:57.613575 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:11:57.613837 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:11:57.613845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:11:57.613915 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.613980 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.614154 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":74,"thread_start_us":95,"threads_started":1}
I20260923 15:11:57.616456 29847 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:48264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 15:11:57.616664 29847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:57.618111 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 1d542ab9323f4e9694ca8559b7222bda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=591f16d8fe0e4052a7b36c2c11775851]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:57.618127 30265 tablet_service.cc:1467] Processing CreateTablet for tablet 1d542ab9323f4e9694ca8559b7222bda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=591f16d8fe0e4052a7b36c2c11775851]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:57.618170 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 1d542ab9323f4e9694ca8559b7222bda (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=591f16d8fe0e4052a7b36c2c11775851]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 15:11:57.618296 30265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d542ab9323f4e9694ca8559b7222bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.618295 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d542ab9323f4e9694ca8559b7222bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.618301 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d542ab9323f4e9694ca8559b7222bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.619396 30360 tablet_bootstrap.cc:492] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: Bootstrap starting.
I20260923 15:11:57.619735 30360 tablet_bootstrap.cc:654] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.619769 30359 tablet_bootstrap.cc:492] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: Bootstrap starting.
I20260923 15:11:57.620151 30359 tablet_bootstrap.cc:654] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.620393 30360 tablet_bootstrap.cc:492] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: No bootstrap required, opened a new log
I20260923 15:11:57.620448 30360 ts_tablet_manager.cc:1397] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260923 15:11:57.620501 30361 tablet_bootstrap.cc:492] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: Bootstrap starting.
I20260923 15:11:57.620568 30360 raft_consensus.cc:348] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.620647 30360 raft_consensus.cc:374] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.620673 30360 raft_consensus.cc:729] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.620728 30360 consensus_queue.cc:260] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.620798 30359 tablet_bootstrap.cc:492] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: No bootstrap required, opened a new log
I20260923 15:11:57.620824 30360 ts_tablet_manager.cc:1428] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.620847 30359 ts_tablet_manager.cc:1397] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:57.620910 30361 tablet_bootstrap.cc:654] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.620857 30380 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.620951 30380 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:57.620988 30359 raft_consensus.cc:348] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.621052 30359 raft_consensus.cc:374] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.621069 30359 raft_consensus.cc:729] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a70824b4844d96a775a5a8dc323476, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.621115 30359 consensus_queue.cc:260] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.621203 30359 ts_tablet_manager.cc:1428] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.621253 30365 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.621330 30365 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:57.621479 30361 tablet_bootstrap.cc:492] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: No bootstrap required, opened a new log
I20260923 15:11:57.621528 30361 ts_tablet_manager.cc:1397] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:57.621690 30361 raft_consensus.cc:348] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.621795 30361 raft_consensus.cc:374] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.621829 30361 raft_consensus.cc:729] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8adf0e64c5461391675fea399f02ba, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.621909 30361 consensus_queue.cc:260] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.622028 30361 ts_tablet_manager.cc:1428] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 15:11:57.622083 30366 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.622184 30366 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:57.635361 30365 raft_consensus.cc:482] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:57.635447 30365 raft_consensus.cc:504] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.635615 30365 leader_election.cc:288] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.635802 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d542ab9323f4e9694ca8559b7222bda" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba" is_pre_election: true
I20260923 15:11:57.635870 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d542ab9323f4e9694ca8559b7222bda" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b4b34b37434d9e978796452bcc2e95" is_pre_election: true
I20260923 15:11:57.635914 30285 raft_consensus.cc:2454] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 0.
I20260923 15:11:57.635950 30005 raft_consensus.cc:2454] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 0.
I20260923 15:11:57.636065 30076 leader_election.cc:302] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8adf0e64c5461391675fea399f02ba, a6a70824b4844d96a775a5a8dc323476; no voters:
I20260923 15:11:57.636148 30365 raft_consensus.cc:2781] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:57.636185 30365 raft_consensus.cc:482] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:57.636215 30365 raft_consensus.cc:3037] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.636825 30365 raft_consensus.cc:504] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.636951 30365 leader_election.cc:288] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 election: Requested vote from peers b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.637100 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d542ab9323f4e9694ca8559b7222bda" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b4b34b37434d9e978796452bcc2e95"
I20260923 15:11:57.637127 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d542ab9323f4e9694ca8559b7222bda" candidate_uuid: "a6a70824b4844d96a775a5a8dc323476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba"
I20260923 15:11:57.637180 30005 raft_consensus.cc:3037] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.637189 30285 raft_consensus.cc:3037] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.637923 30005 raft_consensus.cc:2454] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 1.
I20260923 15:11:57.637923 30285 raft_consensus.cc:2454] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a70824b4844d96a775a5a8dc323476 in term 1.
I20260923 15:11:57.638089 30076 leader_election.cc:302] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8adf0e64c5461391675fea399f02ba, a6a70824b4844d96a775a5a8dc323476; no voters:
I20260923 15:11:57.638181 30365 raft_consensus.cc:2781] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:57.638259 30365 raft_consensus.cc:686] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [term 1 LEADER]: Becoming Leader. State: Replica: a6a70824b4844d96a775a5a8dc323476, State: Running, Role: LEADER
I20260923 15:11:57.638309 30365 consensus_queue.cc:237] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.638518 30390 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: TxnStatusTablet state changed. Reason: New leader a6a70824b4844d96a775a5a8dc323476. Latest consensus state: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.638605 30390 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:57.638801 30402 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 15:11:57.638882 30402 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 15:11:57.638813 29848 catalog_manager.cc:5495] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a70824b4844d96a775a5a8dc323476 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:57.645108 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.642722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.21.204.66:35825)
0923 15:11:57.642797 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.642799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.642806 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:57.642834 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.642835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.642855 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.642920 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:57.643074 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.643556 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.644306 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.644516 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.644589 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.644777 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:57.644879 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:57.644951 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.645022 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.645041 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 15:11:57.645128 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.642515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35825 (local address 127.0.0.1:57230)
0923 15:11:57.642595 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.642606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.642655 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.642951 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.642952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.642953 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:57.643001 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.643004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.643640 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.643641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.644219 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.644221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.644590 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.644591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.644658 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.644674 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:57.644958 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:57.644967 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.645007 (+ 40us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.645054 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 15:11:57.646247 30285 raft_consensus.cc:1261] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Refusing update from remote peer a6a70824b4844d96a775a5a8dc323476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.646332 30005 raft_consensus.cc:1261] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Refusing update from remote peer a6a70824b4844d96a775a5a8dc323476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.646520 30390 consensus_queue.cc:1035] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.646621 30365 consensus_queue.cc:1035] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.647295 30366 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: TxnStatusTablet state changed. Reason: New leader a6a70824b4844d96a775a5a8dc323476. Latest consensus state: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.647387 30366 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:57.647514 30380 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: TxnStatusTablet state changed. Reason: New leader a6a70824b4844d96a775a5a8dc323476. Latest consensus state: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.647601 30380 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:57.647801 30390 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.647878 30390 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:57.648674 30366 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.648758 30366 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P 1c8adf0e64c5461391675fea399f02ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:57.648932 30374 mvcc.cc:205] Tried to move back new op lower bound from 7332562197077159936 to 7332562197046927360. Current Snapshot: MvccSnapshot[applied={T|T < 7332562197077159936 or (T in {7332562197077159936})}]
I20260923 15:11:57.649608 30365 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.649683 30365 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P a6a70824b4844d96a775a5a8dc323476: This TxnStatusTablet replica's current role is: LEADER
I20260923 15:11:57.650359 30380 tablet_replica.cc:439] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a6a70824b4844d96a775a5a8dc323476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } } }
I20260923 15:11:57.650439 30380 tablet_replica.cc:442] T 1d542ab9323f4e9694ca8559b7222bda P b3b4b34b37434d9e978796452bcc2e95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 15:11:57.652099 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.649800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.21.204.66:35825)
0923 15:11:57.649837 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.649839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.649845 (+ 6us) server_negotiation.cc:407] Connection header received
0923 15:11:57.649863 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.649864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.649902 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.649945 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:57.650077 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.650715 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.651449 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.651663 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.651731 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.651870 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:57.651901 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:57.651946 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.652002 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.652019 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:11:57.657124 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.654511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:55885 (local address 127.21.204.66:35825)
0923 15:11:57.654560 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.654562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.654631 (+ 69us) server_negotiation.cc:407] Connection header received
0923 15:11:57.654677 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.654678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.654688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.654753 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:57.654844 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.655545 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.656335 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.656628 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.656700 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.656885 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:57.656916 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:57.656969 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.657014 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.657041 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":32}
I20260923 15:11:57.657125 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.654444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35825 (local address 127.21.204.66:55885)
0923 15:11:57.654573 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:57.654584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:57.654638 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:57.654758 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:57.654758 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:57.654760 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:57.654808 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.654809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.655642 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.655643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.656256 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:57.656258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.656735 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.656736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:57.656793 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.656806 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:57.656975 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:57.656977 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:57.657012 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 15:11:57.657050 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260923 15:11:57.658989 30005 raft_consensus.cc:1261] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Refusing update from remote peer a6a70824b4844d96a775a5a8dc323476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.659068 30285 raft_consensus.cc:1261] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Refusing update from remote peer a6a70824b4844d96a775a5a8dc323476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.659209 30390 consensus_queue.cc:1035] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.659319 30390 consensus_queue.cc:1035] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.673105 29847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:57.673208 29847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:57.673914 29847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Sending DeleteTablet for 3 replicas of tablet b05a71b3c05a490f9ccdf62f2e993fdc
I20260923 15:11:57.674211 30265 tablet_service.cc:1514] Processing DeleteTablet for tablet b05a71b3c05a490f9ccdf62f2e993fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:43208
I20260923 15:11:57.674326 30383 tablet_replica.cc:330] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: stopping tablet replica
I20260923 15:11:57.674382 30383 raft_consensus.cc:2229] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:57.674424 30383 raft_consensus.cc:2258] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.674577 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet b05a71b3c05a490f9ccdf62f2e993fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:57218
I20260923 15:11:57.674679 30384 tablet_replica.cc:330] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: stopping tablet replica
I20260923 15:11:57.674746 30384 raft_consensus.cc:2229] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:57.674832 30384 raft_consensus.cc:2258] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.674919 30383 ts_tablet_manager.cc:1905] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.675108 30384 ts_tablet_manager.cc:1905] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.676795 30384 ts_tablet_manager.cc:1918] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:57.676797 30383 ts_tablet_manager.cc:1918] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:57.676856 30384 log.cc:1196] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/b05a71b3c05a490f9ccdf62f2e993fdc
I20260923 15:11:57.676856 30383 log.cc:1196] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/b05a71b3c05a490f9ccdf62f2e993fdc
I20260923 15:11:57.677088 30383 ts_tablet_manager.cc:1939] T b05a71b3c05a490f9ccdf62f2e993fdc P 1c8adf0e64c5461391675fea399f02ba: Deleting consensus metadata
I20260923 15:11:57.677104 30384 ts_tablet_manager.cc:1939] T b05a71b3c05a490f9ccdf62f2e993fdc P a6a70824b4844d96a775a5a8dc323476: Deleting consensus metadata
I20260923 15:11:57.677397 29832 catalog_manager.cc:4841] TS 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295): tablet b05a71b3c05a490f9ccdf62f2e993fdc (table test-table [id=321f4ceaae1b4685b7ea9e8577223751]) successfully deleted
I20260923 15:11:57.677480 29832 catalog_manager.cc:4841] TS a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825): tablet b05a71b3c05a490f9ccdf62f2e993fdc (table test-table [id=321f4ceaae1b4685b7ea9e8577223751]) successfully deleted
I20260923 15:11:57.678156 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet b05a71b3c05a490f9ccdf62f2e993fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:55528
I20260923 15:11:57.678283 30385 tablet_replica.cc:330] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: stopping tablet replica
I20260923 15:11:57.678359 30385 raft_consensus.cc:2229] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:57.678404 30385 raft_consensus.cc:2258] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.678709 30385 ts_tablet_manager.cc:1905] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.680243 30385 ts_tablet_manager.cc:1918] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:57.680294 30385 log.cc:1196] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/b05a71b3c05a490f9ccdf62f2e993fdc
I20260923 15:11:57.680519 30385 ts_tablet_manager.cc:1939] T b05a71b3c05a490f9ccdf62f2e993fdc P b3b4b34b37434d9e978796452bcc2e95: Deleting consensus metadata
I20260923 15:11:57.680976 29834 catalog_manager.cc:4841] TS b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701): tablet b05a71b3c05a490f9ccdf62f2e993fdc (table test-table [id=321f4ceaae1b4685b7ea9e8577223751]) successfully deleted
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 15:11:57.716455 29898 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:11:57.710214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.21.204.126:42975)
0923 15:11:57.710269 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.710271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.710414 (+ 143us) server_negotiation.cc:407] Connection header received
0923 15:11:57.710504 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.710506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.710518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.710597 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.711307 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.712174 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.713025 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.713228 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.713261 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.716368 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48280: BlockingRecv error: recv got EOF from 127.0.0.1:48280 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 23 15:11:57 dist-test-slave-t5lt krb5kdc[29803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM
I20260923 15:11:57.726649 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.717371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.21.204.126:42975)
0923 15:11:57.717407 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.717409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.717416 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:11:57.717441 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.717442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.717450 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.717561 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:11:57.718239 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.719040 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.720168 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.720435 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.720517 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.723458 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:11:57.723460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:11:57.723460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:11:57.723467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:11:57.725197 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.725652 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.725653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.725654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.725720 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:11:57.725975 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:11:57.725976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:11:57.725976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:11:57.726127 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:11:57.726213 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.726432 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.726514 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:11:57.728471 29847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:57.728672 29847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:57.730121 29985 tablet_service.cc:1467] Processing CreateTablet for tablet ba78e4825ffe41b9b1c1c635a26fd7be (DEFAULT_TABLE table=test-table [id=3da0c3536d92484baab5afe8d0410044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.730268 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba78e4825ffe41b9b1c1c635a26fd7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.732062 30360 tablet_bootstrap.cc:492] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: Bootstrap starting.
I20260923 15:11:57.732174 30125 tablet_service.cc:1467] Processing CreateTablet for tablet ba78e4825ffe41b9b1c1c635a26fd7be (DEFAULT_TABLE table=test-table [id=3da0c3536d92484baab5afe8d0410044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.732321 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba78e4825ffe41b9b1c1c635a26fd7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.732537 30360 tablet_bootstrap.cc:654] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.733215 30360 tablet_bootstrap.cc:492] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: No bootstrap required, opened a new log
I20260923 15:11:57.733296 30360 ts_tablet_manager.cc:1397] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260923 15:11:57.733418 30360 raft_consensus.cc:348] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.733501 30360 raft_consensus.cc:374] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.733529 30360 raft_consensus.cc:729] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.733574 30360 consensus_queue.cc:260] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.733664 30360 ts_tablet_manager.cc:1428] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.733846 30359 tablet_bootstrap.cc:492] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: Bootstrap starting.
I20260923 15:11:57.733958 30265 tablet_service.cc:1467] Processing CreateTablet for tablet ba78e4825ffe41b9b1c1c635a26fd7be (DEFAULT_TABLE table=test-table [id=3da0c3536d92484baab5afe8d0410044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.734076 30265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba78e4825ffe41b9b1c1c635a26fd7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.734265 30359 tablet_bootstrap.cc:654] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.734972 30359 tablet_bootstrap.cc:492] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: No bootstrap required, opened a new log
I20260923 15:11:57.735046 30359 ts_tablet_manager.cc:1397] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:57.735208 30359 raft_consensus.cc:348] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.735272 30359 raft_consensus.cc:374] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.735291 30359 raft_consensus.cc:729] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a70824b4844d96a775a5a8dc323476, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.735337 30359 consensus_queue.cc:260] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.735426 30359 ts_tablet_manager.cc:1428] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.735674 30361 tablet_bootstrap.cc:492] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: Bootstrap starting.
I20260923 15:11:57.736374 30361 tablet_bootstrap.cc:654] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.737339 30361 tablet_bootstrap.cc:492] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: No bootstrap required, opened a new log
I20260923 15:11:57.737397 30361 ts_tablet_manager.cc:1397] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 15:11:57.737555 30361 raft_consensus.cc:348] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.737635 30361 raft_consensus.cc:374] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.737663 30361 raft_consensus.cc:729] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8adf0e64c5461391675fea399f02ba, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.737707 30361 consensus_queue.cc:260] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.737835 30361 ts_tablet_manager.cc:1428] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.850293 30380 raft_consensus.cc:482] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:57.850430 30380 raft_consensus.cc:504] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.850657 30380 leader_election.cc:288] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.850898 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78e4825ffe41b9b1c1c635a26fd7be" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a70824b4844d96a775a5a8dc323476" is_pre_election: true
I20260923 15:11:57.850924 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78e4825ffe41b9b1c1c635a26fd7be" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba" is_pre_election: true
I20260923 15:11:57.851011 30145 raft_consensus.cc:2454] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 0.
I20260923 15:11:57.851025 30285 raft_consensus.cc:2454] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 0.
I20260923 15:11:57.851183 29936 leader_election.cc:302] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a70824b4844d96a775a5a8dc323476, b3b4b34b37434d9e978796452bcc2e95; no voters:
I20260923 15:11:57.851333 30380 raft_consensus.cc:2781] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:57.851384 30380 raft_consensus.cc:482] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:57.851404 30380 raft_consensus.cc:3037] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.852181 30380 raft_consensus.cc:504] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.852324 30380 leader_election.cc:288] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 election: Requested vote from peers a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:57.852473 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78e4825ffe41b9b1c1c635a26fd7be" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a70824b4844d96a775a5a8dc323476"
I20260923 15:11:57.852506 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78e4825ffe41b9b1c1c635a26fd7be" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba"
I20260923 15:11:57.852553 30145 raft_consensus.cc:3037] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.852581 30285 raft_consensus.cc:3037] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:57.853114 30285 raft_consensus.cc:2454] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 1.
I20260923 15:11:57.853302 29936 leader_election.cc:302] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8adf0e64c5461391675fea399f02ba, b3b4b34b37434d9e978796452bcc2e95; no voters:
I20260923 15:11:57.853394 30380 raft_consensus.cc:2781] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:57.853425 30145 raft_consensus.cc:2454] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 1.
I20260923 15:11:57.853437 30380 raft_consensus.cc:686] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 1 LEADER]: Becoming Leader. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Running, Role: LEADER
I20260923 15:11:57.853487 30380 consensus_queue.cc:237] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.854084 29847 catalog_manager.cc:5495] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65). New cstate: current_term: 1 leader_uuid: "b3b4b34b37434d9e978796452bcc2e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:57.884384 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:57.882104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.21.204.65:46701)
0923 15:11:57.882165 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:57.882167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:57.882193 (+ 26us) server_negotiation.cc:407] Connection header received
0923 15:11:57.882220 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:57.882220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:57.882231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:57.882279 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:57.882419 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.882911 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.883685 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:57.883889 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:57.883960 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:57.884108 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:57.884146 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:57.884203 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:57.884273 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 15:11:57.884297 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:11:57.885203 30285 raft_consensus.cc:1261] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Refusing update from remote peer b3b4b34b37434d9e978796452bcc2e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.885373 30145 raft_consensus.cc:1261] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Refusing update from remote peer b3b4b34b37434d9e978796452bcc2e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:57.885478 30380 consensus_queue.cc:1035] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.885573 30367 consensus_queue.cc:1035] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:57.888859 29847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:57.888957 29847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:57.889662 29847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Sending DeleteTablet for 3 replicas of tablet ba78e4825ffe41b9b1c1c635a26fd7be
I20260923 15:11:57.889917 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet ba78e4825ffe41b9b1c1c635a26fd7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:55528
I20260923 15:11:57.890007 30385 tablet_replica.cc:330] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: stopping tablet replica
I20260923 15:11:57.889985 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet ba78e4825ffe41b9b1c1c635a26fd7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:57218
I20260923 15:11:57.890057 30385 raft_consensus.cc:2229] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:57.890082 30265 tablet_service.cc:1514] Processing DeleteTablet for tablet ba78e4825ffe41b9b1c1c635a26fd7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:57 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:43208
I20260923 15:11:57.890117 30385 raft_consensus.cc:2258] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.890174 30383 tablet_replica.cc:330] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: stopping tablet replica
I20260923 15:11:57.890221 30384 tablet_replica.cc:330] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: stopping tablet replica
I20260923 15:11:57.890228 30383 raft_consensus.cc:2229] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:57.890272 30384 raft_consensus.cc:2229] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:57.890277 30383 raft_consensus.cc:2258] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.890309 30384 raft_consensus.cc:2258] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:57.890556 30385 ts_tablet_manager.cc:1905] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.890587 30383 ts_tablet_manager.cc:1905] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.890668 29847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:11:57.890797 29847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:11:57.891851 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 8838e85842254418bfb2f43a32162c71 (DEFAULT_TABLE table=test-table [id=9e326278199a4d2fa17bf4f85c123047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.892130 30265 tablet_service.cc:1467] Processing CreateTablet for tablet 8838e85842254418bfb2f43a32162c71 (DEFAULT_TABLE table=test-table [id=9e326278199a4d2fa17bf4f85c123047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.892242 30385 ts_tablet_manager.cc:1918] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:57.892266 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8838e85842254418bfb2f43a32162c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.893400 30360 tablet_bootstrap.cc:492] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95: Bootstrap starting.
I20260923 15:11:57.893602 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 8838e85842254418bfb2f43a32162c71 (DEFAULT_TABLE table=test-table [id=9e326278199a4d2fa17bf4f85c123047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:11:57.893709 30360 tablet_bootstrap.cc:654] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.893699 30383 ts_tablet_manager.cc:1918] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:57.893709 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8838e85842254418bfb2f43a32162c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.893754 30383 log.cc:1196] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/ba78e4825ffe41b9b1c1c635a26fd7be
I20260923 15:11:57.893970 30265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8838e85842254418bfb2f43a32162c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:11:57.893987 30383 ts_tablet_manager.cc:1939] T ba78e4825ffe41b9b1c1c635a26fd7be P 1c8adf0e64c5461391675fea399f02ba: Deleting consensus metadata
I20260923 15:11:57.892292 30385 log.cc:1196] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/ba78e4825ffe41b9b1c1c635a26fd7be
I20260923 15:11:57.894307 29832 catalog_manager.cc:4841] TS 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295): tablet ba78e4825ffe41b9b1c1c635a26fd7be (table test-table [id=3da0c3536d92484baab5afe8d0410044]) successfully deleted
I20260923 15:11:57.894428 30385 ts_tablet_manager.cc:1939] T ba78e4825ffe41b9b1c1c635a26fd7be P b3b4b34b37434d9e978796452bcc2e95: Deleting consensus metadata
I20260923 15:11:57.894737 29834 catalog_manager.cc:4841] TS b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701): tablet ba78e4825ffe41b9b1c1c635a26fd7be (table test-table [id=3da0c3536d92484baab5afe8d0410044]) successfully deleted
I20260923 15:11:57.895114 30361 tablet_bootstrap.cc:492] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: Bootstrap starting.
I20260923 15:11:57.895166 30384 ts_tablet_manager.cc:1905] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:57.895490 30361 tablet_bootstrap.cc:654] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.896149 30361 tablet_bootstrap.cc:492] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: No bootstrap required, opened a new log
I20260923 15:11:57.896186 30361 ts_tablet_manager.cc:1397] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 15:11:57.896333 30361 raft_consensus.cc:348] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.896412 30361 raft_consensus.cc:374] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.896441 30361 raft_consensus.cc:729] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8adf0e64c5461391675fea399f02ba, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.896451 30384 ts_tablet_manager.cc:1918] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:11:57.896500 30384 log.cc:1196] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/ba78e4825ffe41b9b1c1c635a26fd7be
I20260923 15:11:57.896488 30361 consensus_queue.cc:260] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.896581 30361 ts_tablet_manager.cc:1428] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.896581 30359 tablet_bootstrap.cc:492] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: Bootstrap starting.
I20260923 15:11:57.896700 30384 ts_tablet_manager.cc:1939] T ba78e4825ffe41b9b1c1c635a26fd7be P a6a70824b4844d96a775a5a8dc323476: Deleting consensus metadata
I20260923 15:11:57.896939 30359 tablet_bootstrap.cc:654] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: Neither blocks nor log segments found. Creating new log.
I20260923 15:11:57.897539 29832 catalog_manager.cc:4841] TS a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825): tablet ba78e4825ffe41b9b1c1c635a26fd7be (table test-table [id=3da0c3536d92484baab5afe8d0410044]) successfully deleted
I20260923 15:11:57.897666 30359 tablet_bootstrap.cc:492] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: No bootstrap required, opened a new log
I20260923 15:11:57.897702 30359 ts_tablet_manager.cc:1397] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:11:57.897799 30359 raft_consensus.cc:348] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.897841 30359 raft_consensus.cc:374] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.897854 30359 raft_consensus.cc:729] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a70824b4844d96a775a5a8dc323476, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.897924 30359 consensus_queue.cc:260] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.898016 30359 ts_tablet_manager.cc:1428] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:57.898514 30360 tablet_bootstrap.cc:492] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95: No bootstrap required, opened a new log
I20260923 15:11:57.898561 30360 ts_tablet_manager.cc:1397] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260923 15:11:57.898660 30360 raft_consensus.cc:348] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.898718 30360 raft_consensus.cc:374] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:11:57.898736 30360 raft_consensus.cc:729] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Initialized, Role: FOLLOWER
I20260923 15:11:57.898773 30360 consensus_queue.cc:260] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:57.898854 30360 ts_tablet_manager.cc:1428] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:11:58.052798 30380 raft_consensus.cc:482] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:11:58.052911 30380 raft_consensus.cc:504] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:58.053103 30380 leader_election.cc:288] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:58.053377 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8838e85842254418bfb2f43a32162c71" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba" is_pre_election: true
I20260923 15:11:58.053369 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8838e85842254418bfb2f43a32162c71" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a70824b4844d96a775a5a8dc323476" is_pre_election: true
I20260923 15:11:58.053483 30285 raft_consensus.cc:2454] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 0.
I20260923 15:11:58.053483 30145 raft_consensus.cc:2454] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 0.
I20260923 15:11:58.053683 29936 leader_election.cc:302] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8adf0e64c5461391675fea399f02ba, b3b4b34b37434d9e978796452bcc2e95; no voters:
I20260923 15:11:58.053823 30380 raft_consensus.cc:2781] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:11:58.053911 30380 raft_consensus.cc:482] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:11:58.053943 30380 raft_consensus.cc:3037] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:58.054648 30380 raft_consensus.cc:504] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:58.054778 30380 leader_election.cc:288] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 election: Requested vote from peers a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825), 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295)
I20260923 15:11:58.054940 30145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8838e85842254418bfb2f43a32162c71" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a70824b4844d96a775a5a8dc323476"
I20260923 15:11:58.055023 30145 raft_consensus.cc:3037] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:58.054983 30285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8838e85842254418bfb2f43a32162c71" candidate_uuid: "b3b4b34b37434d9e978796452bcc2e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8adf0e64c5461391675fea399f02ba"
I20260923 15:11:58.055074 30285 raft_consensus.cc:3037] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:11:58.055652 30145 raft_consensus.cc:2454] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 1.
I20260923 15:11:58.055660 30285 raft_consensus.cc:2454] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b4b34b37434d9e978796452bcc2e95 in term 1.
I20260923 15:11:58.055841 29936 leader_election.cc:302] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a70824b4844d96a775a5a8dc323476, b3b4b34b37434d9e978796452bcc2e95; no voters:
I20260923 15:11:58.055946 30380 raft_consensus.cc:2781] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:11:58.056010 30380 raft_consensus.cc:686] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 1 LEADER]: Becoming Leader. State: Replica: b3b4b34b37434d9e978796452bcc2e95, State: Running, Role: LEADER
I20260923 15:11:58.056069 30380 consensus_queue.cc:237] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } }
I20260923 15:11:58.056697 29847 catalog_manager.cc:5495] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65). New cstate: current_term: 1 leader_uuid: "b3b4b34b37434d9e978796452bcc2e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b4b34b37434d9e978796452bcc2e95" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 46701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 } health_report { overall_health: UNKNOWN } } }
I20260923 15:11:58.084232 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:58.082190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:42175 (local address 127.21.204.66:35825)
0923 15:11:58.082263 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:58.082265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:58.082308 (+ 43us) server_negotiation.cc:407] Connection header received
0923 15:11:58.082358 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:58.082359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:58.082368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:58.082410 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:58.082494 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.082971 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.083691 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.083867 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.083919 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:58.084051 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:58.084076 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:58.084112 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:58.084164 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 15:11:58.084180 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 15:11:58.084250 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:58.082130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35825 (local address 127.21.204.65:42175)
0923 15:11:58.082250 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:58.082266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:58.082319 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:58.082411 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:58.082412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:58.082414 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:58.082458 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:58.082459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.083027 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.083028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:58.083597 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:58.083598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.083929 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.083929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:58.083971 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:58.083985 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:58.084121 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:58.084122 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 15:11:58.084156 (+ 34us) client_negotiation.cc:240] Negotiation successful
0923 15:11:58.084192 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":56,"threads_started":1}
I20260923 15:11:58.086126 30145 raft_consensus.cc:1261] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Refusing update from remote peer b3b4b34b37434d9e978796452bcc2e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:58.086122 30285 raft_consensus.cc:1261] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Refusing update from remote peer b3b4b34b37434d9e978796452bcc2e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:11:58.086328 30380 consensus_queue.cc:1035] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a70824b4844d96a775a5a8dc323476" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:58.086439 30367 consensus_queue.cc:1035] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8adf0e64c5461391675fea399f02ba" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:11:58.092854 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:58.090446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:48501 (local address 127.21.204.65:46701)
0923 15:11:58.090482 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 15:11:58.090484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:11:58.090505 (+ 21us) server_negotiation.cc:407] Connection header received
0923 15:11:58.090575 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:11:58.090576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:11:58.090586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:11:58.090628 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:11:58.090763 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.091211 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.091913 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.092226 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.092294 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:58.092538 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:11:58.092588 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:11:58.092687 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 15:11:58.092782 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 15:11:58.092807 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:11:58.092865 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:58.090415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:46701 (local address 127.21.204.66:48501)
0923 15:11:58.090458 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:58.090470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:11:58.090520 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:11:58.090668 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:11:58.090669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:11:58.090670 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 15:11:58.090706 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:58.090708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.091280 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.091281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:58.091825 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:11:58.091827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:11:58.092324 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:11:58.092325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:11:58.092389 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:11:58.092397 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 15:11:58.092737 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 15:11:58.092739 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:11:58.092778 (+ 39us) client_negotiation.cc:240] Negotiation successful
0923 15:11:58.092814 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 15:11:58.098644 29846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:58.098726 29846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:11:58.099403 29846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b2052df1714479ea9d588719e6c408c: Sending DeleteTablet for 3 replicas of tablet 8838e85842254418bfb2f43a32162c71
I20260923 15:11:58.099660 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet 8838e85842254418bfb2f43a32162c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:58 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:55528
I20260923 15:11:58.099737 30265 tablet_service.cc:1514] Processing DeleteTablet for tablet 8838e85842254418bfb2f43a32162c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:58 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:43208
I20260923 15:11:58.099761 30385 tablet_replica.cc:330] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95: stopping tablet replica
I20260923 15:11:58.099833 30385 raft_consensus.cc:2229] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:11:58.099859 30383 tablet_replica.cc:330] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: stopping tablet replica
I20260923 15:11:58.099898 30385 raft_consensus.cc:2258] T 8838e85842254418bfb2f43a32162c71 P b3b4b34b37434d9e978796452bcc2e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:58.099910 30383 raft_consensus.cc:2229] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:58.099944 30383 raft_consensus.cc:2258] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:58.100229 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 8838e85842254418bfb2f43a32162c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:11:58 UTC) from {username='oryx', principal='oryx/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:57218
I20260923 15:11:58.100314 30384 tablet_replica.cc:330] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: stopping tablet replica
I20260923 15:11:58.100335 30383 ts_tablet_manager.cc:1905] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:58.100433 30384 raft_consensus.cc:2229] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:11:58.100486 30384 raft_consensus.cc:2258] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:11:58.100802 30384 ts_tablet_manager.cc:1905] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:11:58.102020 30383 ts_tablet_manager.cc:1918] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:58.102082 30383 log.cc:1196] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/8838e85842254418bfb2f43a32162c71
I20260923 15:11:58.102293 30383 ts_tablet_manager.cc:1939] T 8838e85842254418bfb2f43a32162c71 P 1c8adf0e64c5461391675fea399f02ba: Deleting consensus metadata
I20260923 15:11:58.102319 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 29920
I20260923 15:11:58.102414 30384 ts_tablet_manager.cc:1918] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 15:11:58.102481 30384 log.cc:1196] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/8838e85842254418bfb2f43a32162c71
I20260923 15:11:58.102655 30384 ts_tablet_manager.cc:1939] T 8838e85842254418bfb2f43a32162c71 P a6a70824b4844d96a775a5a8dc323476: Deleting consensus metadata
I20260923 15:11:58.103497 29832 catalog_manager.cc:4841] TS a6a70824b4844d96a775a5a8dc323476 (127.21.204.66:35825): tablet 8838e85842254418bfb2f43a32162c71 (table test-table [id=9e326278199a4d2fa17bf4f85c123047]) successfully deleted
I20260923 15:11:58.104691 29832 catalog_manager.cc:4841] TS 1c8adf0e64c5461391675fea399f02ba (127.21.204.67:43295): tablet 8838e85842254418bfb2f43a32162c71 (table test-table [id=9e326278199a4d2fa17bf4f85c123047]) successfully deleted
W20260923 15:11:58.109165 29834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:46701 (error 108)
I20260923 15:11:58.109391 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 30060
I20260923 15:11:58.111244 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:11:58.111026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:46701 (local address 127.0.0.1:55550)
0923 15:11:58.111143 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 15:11:58.111200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:46701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":47,"threads_started":1}
W20260923 15:11:58.111392 29834 catalog_manager.cc:4584] TS b3b4b34b37434d9e978796452bcc2e95 (127.21.204.65:46701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8838e85842254418bfb2f43a32162c71: Network error: Client connection negotiation failed: client connection to 127.21.204.65:46701: connect: Connection refused (error 111)
I20260923 15:11:58.115471 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 30200
I20260923 15:11:58.119927 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 29816
2026-09-23T15:11:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2688 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:58 dist-test-slave-t5lt krb5kdc[30446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:58 dist-test-slave-t5lt krb5kdc[30446](info): set up 2 sockets
Sep 23 15:11:58 dist-test-slave-t5lt krb5kdc[30446](info): commencing operation
krb5kdc: starting...
W20260923 15:11:59.159802 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:11:59 dist-test-slave-t5lt krb5kdc[30446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:11:59.281267 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33687
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:33687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:11:59.361852 30459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:11:59.362102 30459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:11:59.362139 30459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:11:59.363361 30459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:11:59.363420 30459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:11:59.363443 30459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:11:59.363472 30459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:11:59.363498 30459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:11:59.365026 30459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:33687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:11:59.365296 30459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:11:59.365514 30459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:11:59.370332 30465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:59.370466 30467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:11:59.370565 30464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:11:59.370478 30459 server_base.cc:1034] running on GCE node
I20260923 15:11:59.370886 30459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:11:59.371148 30459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:11:59.372282 30459 hybrid_clock.cc:648] HybridClock initialized: now 1790176319372271 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T15:11:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1257 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:11:59 dist-test-slave-t5lt krb5kdc[30472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:11:59 dist-test-slave-t5lt krb5kdc[30472](info): set up 2 sockets
Sep 23 15:11:59 dist-test-slave-t5lt krb5kdc[30472](info): commencing operation
krb5kdc: starting...
W20260923 15:12:00.416090 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:12:00 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:12:00.538712 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34835
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:00.618816 30485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:00.619012 30485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:00.619035 30485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:00.620149 30485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:12:00.620191 30485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:00.620206 30485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:00.620226 30485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:12:00.620239 30485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:12:00.621728 30485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:00.621974 30485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:00.622191 30485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:00.626857 30491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:00.626847 30493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:00.626847 30490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:00.627106 30485 server_base.cc:1034] running on GCE node
I20260923 15:12:00.627301 30485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:00.627519 30485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:00.628683 30485 hybrid_clock.cc:648] HybridClock initialized: now 1790176320628663 us; error 36 us; skew 500 ppm
Sep 23 15:12:00 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:00.635855 30485 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:12:00.636314 30485 webserver.cc:460] Webserver started at http://127.21.204.126:45025/ using document root <none> and password file <none>
I20260923 15:12:00.636503 30485 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:00.636543 30485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:00.636632 30485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:00.637672 30485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "de4a4521cf4345cc89827ecfd2ce2501"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "c1a23e7b57aa371909f2ccfb09978663"
server_key_iv: "bd8f437321f7a44b447343450655050f"
server_key_version: "encryptionkey@0"
I20260923 15:12:00.638060 30485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "de4a4521cf4345cc89827ecfd2ce2501"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "c1a23e7b57aa371909f2ccfb09978663"
server_key_iv: "bd8f437321f7a44b447343450655050f"
server_key_version: "encryptionkey@0"
I20260923 15:12:00.639379 30485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:00.640138 30500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:00.640332 30485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:00.640419 30485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "de4a4521cf4345cc89827ecfd2ce2501"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "c1a23e7b57aa371909f2ccfb09978663"
server_key_iv: "bd8f437321f7a44b447343450655050f"
server_key_version: "encryptionkey@0"
I20260923 15:12:00.640472 30485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:00.663409 30485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:00.665313 30485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:00.665424 30485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:00.669487 30485 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34835
I20260923 15:12:00.669569 30552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34835 every 8 connection(s)
I20260923 15:12:00.669932 30485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:12:00.670819 30553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:12:00.673144 30553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Bootstrap starting.
I20260923 15:12:00.673177 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 30485
I20260923 15:12:00.673285 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:12:00.673487 22321 external_mini_cluster.cc:1428] Setting key eb8814517d801d3323d8e6d123bdac49
I20260923 15:12:00.673925 30553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Neither blocks nor log segments found. Creating new log.
I20260923 15:12:00.674248 30553 log.cc:824] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:00.674909 30553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: No bootstrap required, opened a new log
I20260923 15:12:00.676676 30553 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:00.676837 30553 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:12:00.676898 30553 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4a4521cf4345cc89827ecfd2ce2501, State: Initialized, Role: FOLLOWER
I20260923 15:12:00.677037 30553 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:00.677115 30553 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:12:00.677160 30553 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:12:00.677206 30553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:12:00.677719 30553 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:00.677845 30553 leader_election.cc:302] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4a4521cf4345cc89827ecfd2ce2501; no voters:
I20260923 15:12:00.678047 30553 leader_election.cc:288] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:12:00.678071 30558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:12:00.678180 30558 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 LEADER]: Becoming Leader. State: Replica: de4a4521cf4345cc89827ecfd2ce2501, State: Running, Role: LEADER
I20260923 15:12:00.678270 30558 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:00.678599 30553 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:12:00.678647 30559 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4a4521cf4345cc89827ecfd2ce2501. Latest consensus state: current_term: 1 leader_uuid: "de4a4521cf4345cc89827ecfd2ce2501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } } }
I20260923 15:12:00.678726 30559 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:00.678949 30562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:12:00.679642 30562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 15:12:00 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:12:00.678586 30558 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4a4521cf4345cc89827ecfd2ce2501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } } }
I20260923 15:12:00.680676 30558 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:00.681319 30562 catalog_manager.cc:1347] Generated new cluster ID: 28853d190dae4a679decf4e367d6d562
I20260923 15:12:00.681371 30562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:12:00.684800 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:00.674026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.21.204.126:34835)
0923 15:12:00.674246 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:00.674260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:00.674285 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:12:00.674758 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:00.674766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:00.674815 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:00.674978 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:00.675619 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:00.676498 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:00.677442 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:00.677869 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:00.678002 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:00.680978 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:00.680981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:00.680982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:00.680995 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:00.683166 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:00.683555 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:00.683559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:00.683559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:00.683668 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:00.683875 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:00.683876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:00.683876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:00.684180 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:00.684332 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:00.684439 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 15:12:00.684576 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260923 15:12:00.716523 30562 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:12:00.717092 30562 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:12:00.723351 30562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Generated new TSK 0
I20260923 15:12:00.723531 30562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:12:00.760450 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34835
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:00.861069 30579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:00.861279 30579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:00.861303 30579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:00.862530 30579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:00.862571 30579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:00.862597 30579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:12:00.864020 30579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:00.864288 30579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:00.864518 30579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:00.870203 30585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:00.870213 30587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:00.870203 30584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:00.870234 30579 server_base.cc:1034] running on GCE node
I20260923 15:12:00.870609 30579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:00.870826 30579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:00.871986 30579 hybrid_clock.cc:648] HybridClock initialized: now 1790176320871967 us; error 33 us; skew 500 ppm
Sep 23 15:12:00 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:00.874861 30579 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:12:00.875449 30579 webserver.cc:460] Webserver started at http://127.21.204.65:40899/ using document root <none> and password file <none>
I20260923 15:12:00.875726 30579 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:00.875780 30579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:00.875913 30579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:00.876977 30579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "b6b34d163cf746caa05e5583710815c9"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "5f85aa41c8f739b59bdc0cee07676f8c"
server_key_iv: "58a016a9c69d3b49a764cd986030dfa9"
server_key_version: "encryptionkey@0"
I20260923 15:12:00.877353 30579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "b6b34d163cf746caa05e5583710815c9"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "5f85aa41c8f739b59bdc0cee07676f8c"
server_key_iv: "58a016a9c69d3b49a764cd986030dfa9"
server_key_version: "encryptionkey@0"
I20260923 15:12:00.878962 30579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:12:00.879904 30594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:00.880136 30579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:12:00.880237 30579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "b6b34d163cf746caa05e5583710815c9"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "5f85aa41c8f739b59bdc0cee07676f8c"
server_key_iv: "58a016a9c69d3b49a764cd986030dfa9"
server_key_version: "encryptionkey@0"
I20260923 15:12:00.880327 30579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:00.909040 30579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:00.911104 30579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:00.911219 30579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:00.911407 30579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:00.911724 30579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:00.911759 30579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:00.911800 30579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:00.911829 30579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:00.917565 30579 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:42351
I20260923 15:12:00.917616 30707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:42351 every 8 connection(s)
I20260923 15:12:00.917934 30579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
Sep 23 15:12:00 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176320, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:12:00.926087 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 30579
I20260923 15:12:00.926215 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:12:00.926432 22321 external_mini_cluster.cc:1428] Setting key 75af806be2dd139fb1f626c42d4d45a6
I20260923 15:12:00.927062 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:00.918710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:49095 (local address 127.21.204.126:34835)
0923 15:12:00.918784 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:00.918787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:00.918981 (+ 194us) server_negotiation.cc:407] Connection header received
0923 15:12:00.919449 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:00.919450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:00.919461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:00.919520 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:00.920454 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:00.920984 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:00.921993 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:00.922205 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:00.922269 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:00.924346 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:00.924347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:00.924347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:00.924354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:00.925525 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:00.925962 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:00.925964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:00.925964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:00.926027 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:00.926305 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:00.926306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:00.926307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:00.926449 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:00.926524 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:00.926892 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 15:12:00.926960 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:12:00.927325 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:00.918673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:49095)
0923 15:12:00.918863 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:00.918899 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:00.919366 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:00.919592 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:00.919595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:00.920170 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:00.920356 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:00.920361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:00.921071 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:00.921072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:00.921846 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:00.921848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:00.922288 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:00.922289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:00.922391 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:00.922829 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:00.922837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:00.924242 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:00.925594 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:00.925601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:00.925603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:00.925838 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:00.926102 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:00.926103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:00.926103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:00.926233 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:00.926522 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:00.926528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:00.926739 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 15:12:00.926870 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 15:12:00.927029 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
I20260923 15:12:00.927974 30708 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34835
I20260923 15:12:00.928076 30708 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:00.928253 30708 heartbeater.cc:511] Master 127.21.204.126:34835 requested a full tablet report, sending...
I20260923 15:12:00.928846 30517 ts_manager.cc:194] Registered new tserver with Master: b6b34d163cf746caa05e5583710815c9 (127.21.204.65:42351)
I20260923 15:12:00.929430 30517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:49095
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:12:00.953248 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34835
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:01.036646 30713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:01.036880 30713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:01.036918 30713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:01.038164 30713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:01.038220 30713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:01.038265 30713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:12:01.039745 30713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:01.040129 30713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:01.040387 30713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:01.046550 30718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.046602 30713 server_base.cc:1034] running on GCE node
W20260923 15:12:01.046550 30719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.046581 30721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.046969 30713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:01.047212 30713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:01.048396 30713 hybrid_clock.cc:648] HybridClock initialized: now 1790176321048357 us; error 57 us; skew 500 ppm
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:01.051309 30713 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:12:01.051806 30713 webserver.cc:460] Webserver started at http://127.21.204.66:36299/ using document root <none> and password file <none>
I20260923 15:12:01.052242 30713 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:01.052342 30713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:01.052497 30713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:01.053547 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "46e9f166993c4a1e8340ca2933966944"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "f9fc1d5ef8504fe8f699422e93549b9a"
server_key_iv: "0114bbc84c34dadbefd0f6d60209a70a"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.053973 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "46e9f166993c4a1e8340ca2933966944"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "f9fc1d5ef8504fe8f699422e93549b9a"
server_key_iv: "0114bbc84c34dadbefd0f6d60209a70a"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.055367 30713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:01.056252 30728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.056458 30713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:01.056557 30713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "46e9f166993c4a1e8340ca2933966944"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "f9fc1d5ef8504fe8f699422e93549b9a"
server_key_iv: "0114bbc84c34dadbefd0f6d60209a70a"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.056635 30713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:01.070269 30713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:01.072553 30713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:01.072714 30713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:01.072945 30713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:01.073269 30713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:01.073323 30713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.073364 30713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:01.073390 30713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.079979 30713 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:37925
I20260923 15:12:01.080034 30841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:37925 every 8 connection(s)
I20260923 15:12:01.080358 30713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:12:01.087904 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 30713
I20260923 15:12:01.088019 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:12:01.088258 22321 external_mini_cluster.cc:1428] Setting key d3d63774d27a65c2dcb36804b97eb1b0
I20260923 15:12:01.090272 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.081099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:45543 (local address 127.21.204.126:34835)
0923 15:12:01.081156 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.081158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.081402 (+ 244us) server_negotiation.cc:407] Connection header received
0923 15:12:01.081985 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.081987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.082000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.082050 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.082989 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.083575 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.084633 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.084836 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.084916 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.087225 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:01.087227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:01.087227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:01.087239 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:01.088878 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:01.089266 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:01.089267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:01.089268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:01.089336 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:01.089586 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:01.089587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:01.089588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:01.089706 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:01.089780 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:01.090103 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 15:12:01.090180 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 15:12:01.090457 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.081074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:45543)
0923 15:12:01.081236 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.081283 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.081840 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.082114 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.082118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.082699 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.082893 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.082898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.083667 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.083668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.084550 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.084553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.084928 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.084929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.085062 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.085675 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.085684 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.087118 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:01.088965 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:01.088972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:01.088974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:01.089194 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:01.089418 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:01.089419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:01.089419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:01.089520 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:01.089771 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:01.089778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:01.089961 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 15:12:01.090073 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 15:12:01.090236 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260923 15:12:01.090904 30842 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34835
I20260923 15:12:01.090993 30842 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:01.091182 30842 heartbeater.cc:511] Master 127.21.204.126:34835 requested a full tablet report, sending...
I20260923 15:12:01.091533 30517 ts_manager.cc:194] Registered new tserver with Master: 46e9f166993c4a1e8340ca2933966944 (127.21.204.66:37925)
I20260923 15:12:01.092046 30517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:45543
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:12:01.113091 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34835
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:01.195252 30847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:01.195464 30847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:01.195488 30847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:01.196595 30847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:01.196631 30847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:01.196657 30847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:12:01.198179 30847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:01.198375 30847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:01.198549 30847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:01.203444 30852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.203471 30853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.203497 30855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.203583 30847 server_base.cc:1034] running on GCE node
I20260923 15:12:01.203888 30847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:01.204126 30847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:01.205276 30847 hybrid_clock.cc:648] HybridClock initialized: now 1790176321205258 us; error 35 us; skew 500 ppm
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:01.207731 30847 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:12:01.208164 30847 webserver.cc:460] Webserver started at http://127.21.204.67:40225/ using document root <none> and password file <none>
I20260923 15:12:01.208397 30847 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:01.208464 30847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:01.208618 30847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:01.209676 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "f69688433c1047a9b3e4ffa7a7a9e56a"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "33b579eec95173932a379624b7202ddd"
server_key_iv: "3b8c1fb45fe338eb803ba75d929bfd71"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.210064 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "f69688433c1047a9b3e4ffa7a7a9e56a"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "33b579eec95173932a379624b7202ddd"
server_key_iv: "3b8c1fb45fe338eb803ba75d929bfd71"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.211357 30847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:01.212024 30862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.212216 30847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:12:01.212316 30847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "f69688433c1047a9b3e4ffa7a7a9e56a"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "33b579eec95173932a379624b7202ddd"
server_key_iv: "3b8c1fb45fe338eb803ba75d929bfd71"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.212385 30847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:01.229001 30847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:01.231019 30847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:01.231179 30847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:01.231408 30847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:01.231725 30847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:01.231775 30847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.231817 30847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:01.231853 30847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.237653 30847 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:37355
I20260923 15:12:01.237740 30975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:37355 every 8 connection(s)
I20260923 15:12:01.238061 30847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:12:01.246960 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.238844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:46699 (local address 127.21.204.126:34835)
0923 15:12:01.238887 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.238889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.239078 (+ 189us) server_negotiation.cc:407] Connection header received
0923 15:12:01.239572 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.239573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.239584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.239647 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.240539 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.241021 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.241984 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.242180 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.242260 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.244369 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:01.244370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:01.244371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:01.244378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:01.245533 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:01.245982 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:01.245984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:01.245984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:01.246045 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:01.246313 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:01.246314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:01.246314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:01.246446 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:01.246517 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:01.246791 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 15:12:01.246870 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:12:01.247133 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.238793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:46699)
0923 15:12:01.238956 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.238998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.239462 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.239713 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.239716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.240276 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.240464 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.240469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.241093 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.241093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.241858 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.241860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.242280 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.242281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.242422 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.242929 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.242938 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.244275 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:01.245614 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:01.245621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:01.245624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:01.245854 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:01.246112 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:01.246113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:01.246113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:01.246227 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:01.246520 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:01.246527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:01.246655 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 15:12:01.246749 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 15:12:01.246910 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260923 15:12:01.247670 30976 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34835
I20260923 15:12:01.247762 30976 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:01.247845 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 30847
I20260923 15:12:01.247962 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:12:01.247970 30976 heartbeater.cc:511] Master 127.21.204.126:34835 requested a full tablet report, sending...
I20260923 15:12:01.248145 22321 external_mini_cluster.cc:1428] Setting key 199f53c4e37b59b9001dbc0e9d0a07f7
I20260923 15:12:01.248399 30517 ts_manager.cc:194] Registered new tserver with Master: f69688433c1047a9b3e4ffa7a7a9e56a (127.21.204.67:37355)
I20260923 15:12:01.248792 30517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:46699
I20260923 15:12:01.248898 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab.
Entry for principal oryx/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab.
WARNING: no policy specified for oryx/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab.
Entry for principal oryx/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab.
WARNING: no policy specified for oryx/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab.
Entry for principal oryx/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab.
I20260923 15:12:01.321106 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 30579
I20260923 15:12:01.325738 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 30713
I20260923 15:12:01.330045 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 30847
I20260923 15:12:01.333952 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 30485
I20260923 15:12:01.339291 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34835
--webserver_interface=127.21.204.126
--webserver_port=45025
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:01.421614 30985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:01.421849 30985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:01.421886 30985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:01.423044 30985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:12:01.423110 30985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:01.423136 30985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:01.423166 30985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:12:01.423197 30985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:12:01.424654 30985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:01.424906 30985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:01.425127 30985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:12:01.430120 30985 server_base.cc:1034] running on GCE node
W20260923 15:12:01.430083 30990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.430114 30991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.430091 30993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.430508 30985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:01.430802 30985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:01.431982 30985 hybrid_clock.cc:648] HybridClock initialized: now 1790176321431981 us; error 41 us; skew 500 ppm
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:01.435124 30985 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:12:01.435616 30985 webserver.cc:460] Webserver started at http://127.21.204.126:45025/ using document root <none> and password file <none>
I20260923 15:12:01.435912 30985 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:01.435997 30985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:01.437467 30985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:01.438279 31000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.438522 30985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:01.438635 30985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "de4a4521cf4345cc89827ecfd2ce2501"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "c1a23e7b57aa371909f2ccfb09978663"
server_key_iv: "bd8f437321f7a44b447343450655050f"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.439028 30985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:01.454572 30985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:01.456647 30985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:01.456823 30985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:01.461014 30985 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34835
I20260923 15:12:01.461064 31052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34835 every 8 connection(s)
I20260923 15:12:01.461436 30985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:12:01.462949 31053 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:12:01.463572 31053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Bootstrap starting.
I20260923 15:12:01.463960 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 30985
I20260923 15:12:01.464253 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:42351
--local_ip_for_outbound_sockets=127.21.204.65
--tserver_master_addrs=127.21.204.126:34835
--webserver_port=40899
--webserver_interface=127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:12:01.472071 31053 log.cc:824] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:01.473685 31053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:12:01.474022 31053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Bootstrap complete.
I20260923 15:12:01.476256 31053 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:01.476498 31053 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4a4521cf4345cc89827ecfd2ce2501, State: Initialized, Role: FOLLOWER
I20260923 15:12:01.476696 31053 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:01.476799 31053 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:12:01.476854 31053 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:12:01.476902 31053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 1 FOLLOWER]: Advancing to term 2
I20260923 15:12:01.477965 31053 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:01.478080 31053 leader_election.cc:302] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4a4521cf4345cc89827ecfd2ce2501; no voters:
I20260923 15:12:01.478246 31053 leader_election.cc:288] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 15:12:01.478312 31057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 15:12:01.478513 31053 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:12:01.478549 31057 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [term 2 LEADER]: Becoming Leader. State: Replica: de4a4521cf4345cc89827ecfd2ce2501, State: Running, Role: LEADER
I20260923 15:12:01.478791 31057 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } }
I20260923 15:12:01.479086 31058 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de4a4521cf4345cc89827ecfd2ce2501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } } }
I20260923 15:12:01.479156 31058 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:01.479439 31059 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4a4521cf4345cc89827ecfd2ce2501. Latest consensus state: current_term: 2 leader_uuid: "de4a4521cf4345cc89827ecfd2ce2501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4a4521cf4345cc89827ecfd2ce2501" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34835 } } }
I20260923 15:12:01.479489 31059 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:01.479637 31064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:12:01.480084 31064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:12:01.480203 31064 catalog_manager.cc:1259] Loaded cluster ID: 28853d190dae4a679decf4e367d6d562
I20260923 15:12:01.485936 31064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:12:01.487156 31064 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:12:01.487362 31064 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de4a4521cf4345cc89827ecfd2ce2501: Loaded TSK: 0
I20260923 15:12:01.487589 31064 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 15:12:01.549600 31055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:01.549795 31055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:01.549818 31055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:01.551055 31055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:01.551097 31055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:01.551123 31055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:12:01.552551 31055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.65.keytab
--principal=oryx/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:42351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=40899
--tserver_master_addrs=127.21.204.126:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:01.552788 31055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:01.553038 31055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:01.558950 31080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.558988 31083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.559149 31055 server_base.cc:1034] running on GCE node
W20260923 15:12:01.558957 31081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.559468 31055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:01.559734 31055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:01.560883 31055 hybrid_clock.cc:648] HybridClock initialized: now 1790176321560872 us; error 27 us; skew 500 ppm
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:01.563566 31055 init.cc:377] Logged in from keytab as oryx/127.21.204.65@KRBTEST.COM (short username oryx)
I20260923 15:12:01.564114 31055 webserver.cc:460] Webserver started at http://127.21.204.65:40899/ using document root <none> and password file <none>
I20260923 15:12:01.564415 31055 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:01.564507 31055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:01.566318 31055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:12:01.567099 31090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.567343 31055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:01.567441 31055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "b6b34d163cf746caa05e5583710815c9"
format_stamp: "Formatted at 2026-09-23 15:12:00 on dist-test-slave-t5lt"
server_key: "5f85aa41c8f739b59bdc0cee07676f8c"
server_key_iv: "58a016a9c69d3b49a764cd986030dfa9"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.567873 31055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:01.593235 31055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:01.595728 31055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:01.595890 31055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:01.596165 31055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:01.596529 31055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:01.596578 31055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.596611 31055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:01.596635 31055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.604041 31055 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:42351
I20260923 15:12:01.604127 31203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:42351 every 8 connection(s)
I20260923 15:12:01.604498 31055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:12:01.609400 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 31055
I20260923 15:12:01.609728 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:37925
--local_ip_for_outbound_sockets=127.21.204.66
--tserver_master_addrs=127.21.204.126:34835
--webserver_port=36299
--webserver_interface=127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.613202 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.605261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:60735)
0923 15:12:01.605434 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.605476 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.606068 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.606520 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.606524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.607130 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.607322 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.607328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.608472 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.608473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.609285 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.609288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.610151 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.610152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.610258 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.610818 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.610827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.612978 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260923 15:12:01.613451 31204 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:34835 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.204.126:34835: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
W20260923 15:12:01.613662 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.605334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:60735 (local address 127.21.204.126:34835)
0923 15:12:01.605600 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.605608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.605649 (+ 41us) server_negotiation.cc:407] Connection header received
0923 15:12:01.606301 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.606306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.606349 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.606483 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.607448 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.608362 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.609375 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.609964 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.610066 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.613123 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:60735: BlockingRecv error: recv got EOF from 127.21.204.65:60735 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.622320 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.613653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:45313)
0923 15:12:01.617954 (+ 4301us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.617969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.618049 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.618229 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.618231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.618405 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.618466 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.618468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.619103 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.619105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.619624 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.619625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.619974 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.619975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.620044 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.620498 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.620503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.622221 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4283,"thread_start_us":78,"threads_started":1}
W20260923 15:12:01.622370 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.613837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:45313 (local address 127.21.204.126:34835)
0923 15:12:01.613939 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.613941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.618042 (+ 4101us) server_negotiation.cc:407] Connection header received
0923 15:12:01.618119 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.618120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.618134 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.618209 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.618542 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.619025 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.619706 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.619902 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.619977 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.622302 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:45313: BlockingRecv error: recv got EOF from 127.21.204.65:45313 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":89,"threads_started":1}
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.627144 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.622886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:32933)
0923 15:12:01.622929 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.622938 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.622973 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.623103 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.623104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.623231 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.623257 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.623259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.623909 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.623910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.624390 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.624392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.624756 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.624757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.624815 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.625229 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.625235 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.627059 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 15:12:01.627477 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.622941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:32933 (local address 127.21.204.126:34835)
0923 15:12:01.622985 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.622987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.622995 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:12:01.623019 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.623020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.623031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.623099 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.623341 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.623820 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.624473 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.624673 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.624754 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.627137 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:32933: BlockingRecv error: recv got EOF from 127.21.204.65:32933 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260923 15:12:01.627463 31204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 15:12:01.697474 31208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:01.697666 31208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:01.697690 31208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:01.698879 31208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:01.698920 31208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:01.698946 31208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:12:01.700742 31208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.66.keytab
--principal=oryx/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:37925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=36299
--tserver_master_addrs=127.21.204.126:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:01.701009 31208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:01.701257 31208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:01.706494 31218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.706511 31216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.706564 31215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.706910 31208 server_base.cc:1034] running on GCE node
I20260923 15:12:01.707151 31208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:01.707422 31208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:01.708572 31208 hybrid_clock.cc:648] HybridClock initialized: now 1790176321708561 us; error 26 us; skew 500 ppm
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:01.711030 31208 init.cc:377] Logged in from keytab as oryx/127.21.204.66@KRBTEST.COM (short username oryx)
I20260923 15:12:01.711504 31208 webserver.cc:460] Webserver started at http://127.21.204.66:36299/ using document root <none> and password file <none>
I20260923 15:12:01.711738 31208 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:01.711818 31208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:01.713182 31208 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:01.713868 31225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.714272 31208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:01.714380 31208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "46e9f166993c4a1e8340ca2933966944"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "f9fc1d5ef8504fe8f699422e93549b9a"
server_key_iv: "0114bbc84c34dadbefd0f6d60209a70a"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.714702 31208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:01.747515 31208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:01.749500 31208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:01.749658 31208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:01.749977 31208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:01.750293 31208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:01.750344 31208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.750384 31208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:01.750412 31208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.756188 31208 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:37925
I20260923 15:12:01.756242 31338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:37925 every 8 connection(s)
I20260923 15:12:01.756553 31208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 15:12:01.764240 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.757305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:42497 (local address 127.21.204.126:34835)
0923 15:12:01.757373 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.757375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.757584 (+ 209us) server_negotiation.cc:407] Connection header received
0923 15:12:01.758187 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.758188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.758206 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.758252 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.759209 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.759706 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.761018 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.761204 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.761272 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.764183 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:42497: BlockingRecv error: recv got EOF from 127.21.204.66:42497 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:12:01.764312 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.757302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:42497)
0923 15:12:01.757474 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.757508 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.758072 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.758340 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.758344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.758923 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.759110 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.759114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.759797 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.759798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.760946 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.760948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.761300 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.761301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.761430 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.761925 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.761935 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.764110 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
W20260923 15:12:01.764523 31339 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:34835 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.204.126:34835: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
I20260923 15:12:01.765843 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 31208
I20260923 15:12:01.766155 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:37355
--local_ip_for_outbound_sockets=127.21.204.67
--tserver_master_addrs=127.21.204.126:34835
--webserver_port=40225
--webserver_interface=127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.769116 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.764758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:33967)
0923 15:12:01.764806 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.764819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.764862 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.764989 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.764990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.765091 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.765130 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.765131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.765695 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.765696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.766549 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.766552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.766913 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.766915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.766981 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.767346 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.767352 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.769075 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 15:12:01.769259 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.764828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:33967 (local address 127.21.204.126:34835)
0923 15:12:01.764879 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.764881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.764888 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:12:01.764912 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.764912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.764920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.764966 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.765183 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.765618 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.766668 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.766848 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.766898 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.769176 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:33967: BlockingRecv error: recv got EOF from 127.21.204.66:33967 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.774039 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.769473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:54695)
0923 15:12:01.769579 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.769588 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.769633 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.769749 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.769750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.769929 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.769956 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.769958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.770891 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.770891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.771341 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.771342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.771832 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.771833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.771882 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.772297 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.772301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.773996 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
W20260923 15:12:01.774156 31339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 15:12:01.774170 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.769536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:54695 (local address 127.21.204.126:34835)
0923 15:12:01.769571 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.769573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.769641 (+ 68us) server_negotiation.cc:407] Connection header received
0923 15:12:01.769679 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.769680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.769690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.769741 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.770029 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.770815 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.771417 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.771736 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.771834 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.774094 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:54695: BlockingRecv error: recv got EOF from 127.21.204.66:54695 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 15:12:01.848994 31342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:01.849186 31342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:01.849210 31342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:01.850373 31342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:01.850420 31342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:01.850446 31342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:12:01.851792 31342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/krb5kdc/oryx_127.21.204.67.keytab
--principal=oryx/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:37355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=40225
--tserver_master_addrs=127.21.204.126:34835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:01.851972 31342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:01.852133 31342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:01.856989 31349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:01.856995 31348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.857079 31342 server_base.cc:1034] running on GCE node
W20260923 15:12:01.856995 31351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:01.857399 31342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:01.857581 31342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:01.858728 31342 hybrid_clock.cc:648] HybridClock initialized: now 1790176321858708 us; error 34 us; skew 500 ppm
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176321, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:01.861029 31342 init.cc:377] Logged in from keytab as oryx/127.21.204.67@KRBTEST.COM (short username oryx)
I20260923 15:12:01.861480 31342 webserver.cc:460] Webserver started at http://127.21.204.67:40225/ using document root <none> and password file <none>
I20260923 15:12:01.861688 31342 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:01.861760 31342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:01.863075 31342 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:01.863616 31358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.863794 31342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:12:01.863880 31342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "f69688433c1047a9b3e4ffa7a7a9e56a"
format_stamp: "Formatted at 2026-09-23 15:12:01 on dist-test-slave-t5lt"
server_key: "33b579eec95173932a379624b7202ddd"
server_key_iv: "3b8c1fb45fe338eb803ba75d929bfd71"
server_key_version: "encryptionkey@0"
I20260923 15:12:01.864187 31342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:01.879097 31342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:01.880877 31342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:01.881021 31342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:01.881234 31342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:01.881531 31342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:01.881563 31342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.881583 31342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:01.881596 31342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:01.887329 31342 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:37355
I20260923 15:12:01.887378 31471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:37355 every 8 connection(s)
I20260923 15:12:01.887725 31342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:12:01.891587 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 31342
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.895169 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.888428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:40817)
0923 15:12:01.888581 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.888621 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.889077 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.889274 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.889277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.889838 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.890092 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.890097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.890732 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.890733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.891654 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.891657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.892218 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.892219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.892347 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.892837 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.892846 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.894994 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260923 15:12:01.895161 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.888462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:40817 (local address 127.21.204.126:34835)
0923 15:12:01.888511 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.888513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.888703 (+ 190us) server_negotiation.cc:407] Connection header received
0923 15:12:01.889156 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.889157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.889167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.889221 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.890170 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.890659 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.891835 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.892141 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.892210 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.895090 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:40817: BlockingRecv error: recv got EOF from 127.21.204.67:40817 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 15:12:01.895435 31472 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:34835 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.204.126:34835: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.898947 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.892132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.21.204.126:34835)
0923 15:12:01.892189 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.892191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.892200 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:12:01.892270 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.892271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.892282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.892363 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.892944 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.893424 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.894304 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.894528 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.894597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.895970 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:01.895975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:01.895975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:01.895986 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:01.897579 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:01.897979 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:01.897981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:01.897982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:01.898078 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:01.898281 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:01.898282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:01.898282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:01.898494 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:01.898639 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:01.898747 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 15:12:01.898860 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.899919 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.895654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:59693)
0923 15:12:01.895699 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.895709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.895747 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.895863 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.895864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.895984 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.896022 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.896023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.896628 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.896629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.897118 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.897120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.897482 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.897483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.897550 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.897956 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.897961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.899869 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 15:12:01.900040 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.895712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:59693 (local address 127.21.204.126:34835)
0923 15:12:01.895766 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.895769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.895777 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:12:01.895797 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.895798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.895807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.895850 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.896100 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.896550 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.897198 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.897401 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.897479 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.899970 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:59693: BlockingRecv error: recv got EOF from 127.21.204.67:59693 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:01 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:01.903680 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:01.900282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:56985)
0923 15:12:01.900323 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:01.900334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:01.900376 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:01.900471 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:01.900472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:01.900570 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:01.900593 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.900594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.901095 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.901096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.901541 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:01.901542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.901826 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.901827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:01.901862 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.902206 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:01.902210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:01.903640 (+ 1430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 15:12:01.903788 31207 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:01.900341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:56985 (local address 127.21.204.126:34835)
0923 15:12:01.900376 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:01.900377 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:01.900385 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:12:01.900423 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:01.900424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:01.900430 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:01.900469 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:01.900634 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.901043 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.901606 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:01.901768 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:01.901816 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:01.903717 (+ 1901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:56985: BlockingRecv error: recv got EOF from 127.21.204.67:56985 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 15:12:01.903829 31472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 15:12:02 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:02 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:02.632515 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:02.627842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:48979)
0923 15:12:02.627967 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:02.627978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:02.628032 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:02.628213 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:02.628214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:02.628378 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:02.628431 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:02.628433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.629087 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.629088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:02.629660 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:02.629662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.630089 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.630090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:02.630132 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:02.630602 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:02.630607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:02.632408 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
W20260923 15:12:02.632560 31477 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:02.627959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:48979 (local address 127.21.204.126:34835)
0923 15:12:02.628087 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:02.628089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:02.628097 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:12:02.628120 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:02.628120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:02.628130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:02.628199 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:02.628502 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.629014 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.629753 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.630023 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.630085 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:02.632488 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:48979: BlockingRecv error: recv got EOF from 127.21.204.65:48979 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
Sep 23 15:12:02 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:02 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:02.779563 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:02.774498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:46811)
0923 15:12:02.774634 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:02.774646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:02.774709 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:02.774903 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:02.774904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:02.775087 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:02.775179 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:02.775181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.776087 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.776088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:02.776639 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:02.776641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.776940 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.776941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:02.776981 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:02.777458 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:02.777468 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:02.779452 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260923 15:12:02.779592 31477 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:02.774574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:46811 (local address 127.21.204.126:34835)
0923 15:12:02.774636 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:02.774638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:02.774725 (+ 87us) server_negotiation.cc:407] Connection header received
0923 15:12:02.774765 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:02.774766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:02.774787 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:02.774846 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:02.775265 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.776021 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.776704 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.776883 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.776931 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:02.779527 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:46811: BlockingRecv error: recv got EOF from 127.21.204.66:46811 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 23 15:12:02 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:02 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:02.909221 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:02.904202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:53827)
0923 15:12:02.904378 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:02.904391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:02.904467 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:02.904603 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:02.904603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:02.904779 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:02.904849 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:02.904852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.905525 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.905526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:02.906111 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:02.906113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.906575 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.906576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:02.906655 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:02.907196 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:02.907202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:02.909151 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
W20260923 15:12:02.909396 31477 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:02.904305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:53827 (local address 127.21.204.126:34835)
0923 15:12:02.904382 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:02.904385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:02.904445 (+ 60us) server_negotiation.cc:407] Connection header received
0923 15:12:02.904524 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:02.904525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:02.904540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:02.904596 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:02.904932 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.905443 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.906211 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:02.906476 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:02.906554 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:02.909292 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:53827: BlockingRecv error: recv got EOF from 127.21.204.67:53827 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 15:12:03 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:03 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:03.638017 31480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:03.633029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:55089)
0923 15:12:03.633255 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:03.633266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:03.633332 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:03.633476 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:03.633476 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:03.633649 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:03.633702 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:03.633703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.634438 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.634439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:03.634964 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:03.634966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.635342 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.635344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:03.635394 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:03.635933 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:03.635938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:03.637958 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":140,"threads_started":1}
W20260923 15:12:03.638218 31481 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:03.633150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:55089 (local address 127.21.204.126:34835)
0923 15:12:03.633269 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:03.633271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:03.633335 (+ 64us) server_negotiation.cc:407] Connection header received
0923 15:12:03.633379 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:03.633380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:03.633391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:03.633459 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:03.633799 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.634364 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.635037 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.635239 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.635291 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:03.638098 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:55089: BlockingRecv error: recv got EOF from 127.21.204.65:55089 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
Sep 23 15:12:03 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:03 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:03.784914 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:03.780055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:44723)
0923 15:12:03.780250 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:03.780265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:03.780328 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:03.780538 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:03.780539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:03.780718 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:03.780767 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:03.780768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.781448 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.781449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:03.782061 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:03.782064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.782440 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.782441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:03.782501 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:03.782964 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:03.782969 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:03.784851 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":136,"threads_started":1}
W20260923 15:12:03.785104 31481 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:03.780137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:44723 (local address 127.21.204.126:34835)
0923 15:12:03.780214 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:03.780217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:03.780349 (+ 132us) server_negotiation.cc:407] Connection header received
0923 15:12:03.780378 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:03.780379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:03.780411 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:03.780502 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:03.780853 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.781357 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.782147 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.782349 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.782437 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:03.784994 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:44723: BlockingRecv error: recv got EOF from 127.21.204.66:44723 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 15:12:03 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:03 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:03.916592 31483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:03.909781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:60283)
0923 15:12:03.909978 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:03.909987 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:03.910082 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:03.910270 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:03.910271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:03.910526 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:03.910603 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:03.910606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.911736 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.911737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:03.912779 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:03.912795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.913248 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.913249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:03.913309 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:03.913952 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:03.913959 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:03.916532 (+ 2573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
W20260923 15:12:03.916713 31481 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:03.909841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:60283 (local address 127.21.204.126:34835)
0923 15:12:03.909907 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:03.909909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:03.910062 (+ 153us) server_negotiation.cc:407] Connection header received
0923 15:12:03.910163 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:03.910165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:03.910190 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:03.910252 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:03.910669 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.911610 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.912867 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:03.913150 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:03.913228 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:03.916629 (+ 3401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:60283: BlockingRecv error: recv got EOF from 127.21.204.67:60283 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 23 15:12:04 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:04 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:04.643342 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:04.638499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:49343)
0923 15:12:04.638638 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:04.638651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:04.638721 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:04.638903 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:04.638904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:04.639094 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:04.639150 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:04.639152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.639929 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.639931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:04.640492 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:04.640494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.640858 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.640859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:04.640910 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:04.641447 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:04.641452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:04.643282 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260923 15:12:04.643553 31485 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:04.638576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:49343 (local address 127.21.204.126:34835)
0923 15:12:04.638740 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:04.638742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:04.638751 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:12:04.638775 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:04.638775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:04.638786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:04.638855 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:04.639228 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.639839 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.640565 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.640778 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.640829 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:04.643407 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:49343: BlockingRecv error: recv got EOF from 127.21.204.65:49343 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
Sep 23 15:12:04 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:04 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:04.789844 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:04.785455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:58279)
0923 15:12:04.785581 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:04.785592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:04.785647 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:04.785781 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:04.785781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:04.785958 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:04.786008 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:04.786010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.786695 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.786696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:04.787201 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:04.787203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.787580 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.787580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:04.787619 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:04.788028 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:04.788032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:04.789786 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260923 15:12:04.789994 31485 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:04.785523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:58279 (local address 127.21.204.126:34835)
0923 15:12:04.785601 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:04.785603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:04.785637 (+ 34us) server_negotiation.cc:407] Connection header received
0923 15:12:04.785708 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:04.785709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:04.785721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:04.785776 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:04.786095 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.786601 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.787302 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.787494 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.787569 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:04.789934 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:58279: BlockingRecv error: recv got EOF from 127.21.204.66:58279 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 15:12:04 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:04 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:04.927856 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:04.917218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:38139)
0923 15:12:04.917353 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:04.917395 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:04.917487 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:04.917693 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:04.917694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:04.917968 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:04.918027 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:04.918030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.918792 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.918794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:04.919381 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:04.919383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.919780 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.919781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:04.919847 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:04.925953 (+ 6106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:04.925962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:04.927752 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260923 15:12:04.927904 31485 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:04.917353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:38139 (local address 127.21.204.126:34835)
0923 15:12:04.917436 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:04.917439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:04.917469 (+ 30us) server_negotiation.cc:407] Connection header received
0923 15:12:04.917586 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:04.917592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:04.917608 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:04.917681 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:04.918122 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.918690 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.919473 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:04.919697 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:04.919781 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:04.927838 (+ 8057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:38139: BlockingRecv error: recv got EOF from 127.21.204.67:38139 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 15:12:05 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:05 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:05.652014 31488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:05.643896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:55845)
0923 15:12:05.644040 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:05.644051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:05.644113 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:05.644261 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:05.644262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:05.644433 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:05.644484 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:05.644485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.645140 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.645141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:05.645655 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:05.645657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.646039 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.646040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:05.646080 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:05.649754 (+ 3674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:05.649759 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:05.651950 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260923 15:12:05.652153 31489 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:05.643950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:55845 (local address 127.21.204.126:34835)
0923 15:12:05.644082 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:05.644084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:05.644094 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:12:05.644168 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:05.644168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:05.644178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:05.644243 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:05.644559 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.645074 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.645722 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.645978 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.646032 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:05.652059 (+ 6027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:55845: BlockingRecv error: recv got EOF from 127.21.204.65:55845 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
Sep 23 15:12:05 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:05 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:05.795975 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:05.790457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:33005)
0923 15:12:05.790612 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:05.790623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:05.790672 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:05.790799 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:05.790799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:05.790966 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:05.791015 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:05.791016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.791785 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.791786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:05.792466 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:05.792470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.792992 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.792993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:05.793128 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:05.793830 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:05.793835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:05.795913 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260923 15:12:05.796151 31489 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:05.790539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:33005 (local address 127.21.204.126:34835)
0923 15:12:05.790633 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:05.790670 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:05.790682 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:12:05.790714 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:05.790715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:05.790726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:05.790788 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:05.791117 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.791682 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.792613 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.792868 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.792974 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:05.796036 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:33005: BlockingRecv error: recv got EOF from 127.21.204.66:33005 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 15:12:05 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:05 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:05.933126 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:05.928420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:42853)
0923 15:12:05.928569 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:05.928579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:05.928634 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:05.928825 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:05.928826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:05.928993 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:05.929046 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:05.929048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.929722 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.929723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:05.930370 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:05.930372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.930709 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.930710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:05.930752 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:05.931274 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:05.931279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:05.933021 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
W20260923 15:12:05.933174 31489 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:05.928485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:42853 (local address 127.21.204.126:34835)
0923 15:12:05.928559 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:05.928561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:05.928633 (+ 72us) server_negotiation.cc:407] Connection header received
0923 15:12:05.928731 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:05.928732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:05.928744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:05.928804 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:05.929105 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.929624 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.930443 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:05.930624 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:05.930685 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:05.933097 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:42853: BlockingRecv error: recv got EOF from 127.21.204.67:42853 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 15:12:06 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:06 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:06.656982 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:06.652505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:50257)
0923 15:12:06.652655 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:06.652668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:06.652744 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:06.652885 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:06.652887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:06.653050 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:06.653101 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:06.653103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.653742 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.653743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:06.654286 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:06.654288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.654619 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.654620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:06.654658 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:06.655085 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:06.655090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:06.656924 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":51,"threads_started":1}
W20260923 15:12:06.657127 31493 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:06.652609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:50257 (local address 127.21.204.126:34835)
0923 15:12:06.652744 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:06.652746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:06.652754 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:12:06.652796 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:06.652797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:06.652807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:06.652863 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:06.653170 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.653668 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.654359 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.654549 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.654601 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:06.657034 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:50257: BlockingRecv error: recv got EOF from 127.21.204.65:50257 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
Sep 23 15:12:06 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:06 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:06.801213 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:06.796501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:36355)
0923 15:12:06.796653 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:06.796668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:06.796736 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:06.796921 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:06.796922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:06.797099 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:06.797147 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:06.797149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.797835 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.797836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:06.798395 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:06.798397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.798806 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.798807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:06.798872 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:06.799363 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:06.799368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:06.801120 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
W20260923 15:12:06.801271 31493 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:06.796623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:36355 (local address 127.21.204.126:34835)
0923 15:12:06.796682 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:06.796684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:06.796718 (+ 34us) server_negotiation.cc:407] Connection header received
0923 15:12:06.796824 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:06.796825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:06.796837 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:06.796912 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:06.797232 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.797734 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.798481 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.798709 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.798803 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:06.801209 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:36355: BlockingRecv error: recv got EOF from 127.21.204.66:36355 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 23 15:12:06 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:06 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:06.938282 31495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:06.933612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:37079)
0923 15:12:06.933740 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:06.933751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:06.933798 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:06.934003 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:06.934003 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:06.934163 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:06.934214 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:06.934216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.934946 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.934947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:06.935490 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:06.935493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.935909 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.935910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:06.935954 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:06.936398 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:06.936402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:06.938219 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260923 15:12:06.938446 31493 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:06.933697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:37079 (local address 127.21.204.126:34835)
0923 15:12:06.933790 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:06.933792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:06.933804 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:12:06.933906 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:06.933907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:06.933921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:06.933986 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:06.934282 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.934862 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.935627 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:06.935828 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:06.935892 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:06.938352 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:37079: BlockingRecv error: recv got EOF from 127.21.204.67:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 15:12:07 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:07 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:07.662191 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:07.657470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:60499)
0923 15:12:07.657638 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:07.657650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:07.657709 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:07.657851 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:07.657852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:07.658037 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:07.658085 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:07.658087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.658781 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.658782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:07.659310 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:07.659312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.659788 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.659789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:07.659842 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:07.660295 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:07.660300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:07.662134 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
W20260923 15:12:07.662333 31497 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:07.657565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:60499 (local address 127.21.204.126:34835)
0923 15:12:07.657665 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:07.657667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:07.657695 (+ 28us) server_negotiation.cc:407] Connection header received
0923 15:12:07.657777 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:07.657777 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:07.657788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:07.657846 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:07.658153 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.658709 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.659408 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.659663 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.659735 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:07.662239 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:60499: BlockingRecv error: recv got EOF from 127.21.204.65:60499 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
Sep 23 15:12:07 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:07 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:07.806715 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:07.801792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:47745)
0923 15:12:07.802006 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:07.802017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:07.802082 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:07.802210 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:07.802211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:07.802379 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:07.802429 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:07.802431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.803146 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.803147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:07.803696 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:07.803697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.804080 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.804081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:07.804137 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:07.804592 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:07.804596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:07.806616 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":154,"threads_started":1}
W20260923 15:12:07.806772 31497 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:07.801851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:47745 (local address 127.21.204.126:34835)
0923 15:12:07.801929 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:07.801931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:07.802085 (+ 154us) server_negotiation.cc:407] Connection header received
0923 15:12:07.802136 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:07.802136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:07.802148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:07.802199 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:07.802527 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.803041 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.803790 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.803988 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.804047 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:07.806705 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:47745: BlockingRecv error: recv got EOF from 127.21.204.66:47745 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 15:12:07 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:07 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:07.944018 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:07.938884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:49431)
0923 15:12:07.939056 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:07.939073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:07.939149 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:07.939369 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:07.939371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:07.939534 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:07.939670 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:07.939674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.940477 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.940478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:07.941012 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:07.941014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.941487 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.941488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:07.941534 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:07.942023 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:07.942027 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:07.943918 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
W20260923 15:12:07.944056 31497 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:07.939012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:49431 (local address 127.21.204.126:34835)
0923 15:12:07.939123 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:07.939125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:07.939138 (+ 13us) server_negotiation.cc:407] Connection header received
0923 15:12:07.939217 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:07.939229 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:07.939246 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:07.939305 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:07.939785 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.940366 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.941113 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:07.941399 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:07.941462 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:07.943995 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:49431: BlockingRecv error: recv got EOF from 127.21.204.67:49431 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 15:12:08 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:08 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:08.667579 31500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:08.662712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:45727)
0923 15:12:08.662886 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:08.662901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:08.662957 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:08.663095 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:08.663096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:08.663259 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:08.663310 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:08.663312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.663966 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.663967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:08.664515 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:08.664516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.664949 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.664950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:08.665002 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:08.665454 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:08.665459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:08.667460 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":114,"threads_started":1}
W20260923 15:12:08.667618 31501 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:08.662761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:45727 (local address 127.21.204.126:34835)
0923 15:12:08.662889 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:08.662891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:08.662955 (+ 64us) server_negotiation.cc:407] Connection header received
0923 15:12:08.663012 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:08.663013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:08.663024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:08.663073 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:08.663375 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.663889 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.664616 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.664843 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.664915 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:08.667548 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:45727: BlockingRecv error: recv got EOF from 127.21.204.65:45727 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
Sep 23 15:12:08 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:08 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:08.812409 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:08.807390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:45761)
0923 15:12:08.807617 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:08.807632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:08.807738 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:08.807871 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:08.807872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:08.808058 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:08.808106 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:08.808108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.808808 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.808809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:08.809351 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:08.809353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.809737 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.809738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:08.809796 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:08.810351 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:08.810356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:08.812315 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
W20260923 15:12:08.812474 31501 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:08.807514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:45761 (local address 127.21.204.126:34835)
0923 15:12:08.807622 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:08.807625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:08.807718 (+ 93us) server_negotiation.cc:407] Connection header received
0923 15:12:08.807797 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:08.807798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:08.807812 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:08.807867 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:08.808198 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.808717 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.809455 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.809662 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.809739 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:08.812409 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:45761: BlockingRecv error: recv got EOF from 127.21.204.66:45761 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 15:12:08 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:08 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:08.949452 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:08.944586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:37573)
0923 15:12:08.944749 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:08.944760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:08.944823 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:08.944983 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:08.944984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:08.945166 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:08.945227 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:08.945229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.945933 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.945934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:08.946473 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:08.946475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.946868 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.946869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:08.946930 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:08.947430 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:08.947435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:08.949353 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
W20260923 15:12:08.949503 31501 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:08.944654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:37573 (local address 127.21.204.126:34835)
0923 15:12:08.944739 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:08.944741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:08.944806 (+ 65us) server_negotiation.cc:407] Connection header received
0923 15:12:08.944876 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:08.944877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:08.944891 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:08.944974 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:08.945310 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.945818 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.946562 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:08.946780 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:08.946865 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:08.949437 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:37573: BlockingRecv error: recv got EOF from 127.21.204.67:37573 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 15:12:09 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:09 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:09.672719 31504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:09.668044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:53923)
0923 15:12:09.668173 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:09.668184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:09.668257 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:09.668425 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:09.668426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:09.668596 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:09.668647 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:09.668649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.669333 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.669338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:09.669914 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:09.669916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.670289 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.670291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:09.670350 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:09.670775 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:09.670779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:09.672663 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260923 15:12:09.672823 31505 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:09.668152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:53923 (local address 127.21.204.126:34835)
0923 15:12:09.668283 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:09.668285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:09.668305 (+ 20us) server_negotiation.cc:407] Connection header received
0923 15:12:09.668328 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:09.668329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:09.668340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:09.668407 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:09.668725 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.669243 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.670001 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.670211 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.670290 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:09.672751 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:53923: BlockingRecv error: recv got EOF from 127.21.204.65:53923 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
Sep 23 15:12:09 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:09 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:09.818089 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:09.812987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:35387)
0923 15:12:09.813167 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:09.813180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:09.813254 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:09.813393 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:09.813394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:09.813553 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:09.813603 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:09.813605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.814488 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.814489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:09.815045 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:09.815047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.815436 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.815436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:09.815479 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:09.815957 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:09.815962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:09.818019 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
W20260923 15:12:09.818298 31505 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:09.813097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:35387 (local address 127.21.204.126:34835)
0923 15:12:09.813172 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:09.813173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:09.813261 (+ 88us) server_negotiation.cc:407] Connection header received
0923 15:12:09.813313 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:09.813314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:09.813324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:09.813370 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:09.813701 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.814394 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.815151 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.815357 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.815428 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:09.818219 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:35387: BlockingRecv error: recv got EOF from 127.21.204.66:35387 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 23 15:12:09 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:09 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:09.954803 31507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:09.950081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:36391)
0923 15:12:09.950233 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:09.950245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:09.950336 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:09.950468 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:09.950469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:09.950647 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:09.950694 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:09.950696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.951409 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.951410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:09.951992 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:09.951995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.952374 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.952375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:09.952439 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:09.952881 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:09.952885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:09.954675 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
W20260923 15:12:09.954811 31505 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:09.950175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:36391 (local address 127.21.204.126:34835)
0923 15:12:09.950223 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:09.950226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:09.950322 (+ 96us) server_negotiation.cc:407] Connection header received
0923 15:12:09.950393 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:09.950394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:09.950409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:09.950454 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:09.950813 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.951318 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.952080 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:09.952285 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:09.952370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:09.954757 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:36391: BlockingRecv error: recv got EOF from 127.21.204.67:36391 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 15:12:10 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:10 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:10.682775 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:10.673215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:39723)
0923 15:12:10.673370 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:10.673380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:10.673431 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:10.673600 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:10.673601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:10.673776 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:10.673826 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:10.673828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.674508 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.674509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:10.675351 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:10.675354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.675671 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.675672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:10.675711 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:10.680763 (+ 5052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:10.680769 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:10.682701 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260923 15:12:10.682919 31509 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:10.673299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:39723 (local address 127.21.204.126:34835)
0923 15:12:10.673422 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:10.673424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:10.673433 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:12:10.673477 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:10.673478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:10.673489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:10.673554 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:10.673934 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.674440 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.675417 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.675608 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.675660 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:10.682821 (+ 7161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:39723: BlockingRecv error: recv got EOF from 127.21.204.65:39723 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
Sep 23 15:12:10 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:10 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:10.823552 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:10.818636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:49245)
0923 15:12:10.818827 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:10.818839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:10.818913 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:10.819067 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:10.819068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:10.819252 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:10.819312 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:10.819314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.820014 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.820015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:10.820553 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:10.820555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.820941 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.820942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:10.821019 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:10.821494 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:10.821499 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:10.823489 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
W20260923 15:12:10.823622 31509 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:10.818734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:49245 (local address 127.21.204.126:34835)
0923 15:12:10.818818 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:10.818821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:10.818891 (+ 70us) server_negotiation.cc:407] Connection header received
0923 15:12:10.818973 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:10.818974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:10.818988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:10.819059 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:10.819394 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.819921 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.820640 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.820855 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.820932 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:10.823575 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:49245: BlockingRecv error: recv got EOF from 127.21.204.66:49245 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 15:12:10 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:10 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:10.960284 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:10.955369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:47915)
0923 15:12:10.955513 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:10.955528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:10.955599 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:10.955722 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:10.955723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:10.955887 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:10.955935 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:10.955937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.956632 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.956633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:10.957163 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:10.957166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.957543 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.957544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:10.957595 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:10.958246 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:10.958252 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:10.960227 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1}
W20260923 15:12:10.960446 31509 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:10.955461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:47915 (local address 127.21.204.126:34835)
0923 15:12:10.955514 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:10.955516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:10.955584 (+ 68us) server_negotiation.cc:407] Connection header received
0923 15:12:10.955651 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:10.955652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:10.955666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:10.955705 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:10.956011 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.956515 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.957256 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:10.957453 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:10.957516 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:10.960354 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:47915: BlockingRecv error: recv got EOF from 127.21.204.67:47915 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 23 15:12:11 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:11 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:11.688330 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:11.683269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:41313)
0923 15:12:11.683624 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:11.683636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:11.683711 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:11.683880 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:11.683881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:11.684050 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:11.684100 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:11.684102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.684781 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.684782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:11.685300 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:11.685301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.685752 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.685753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:11.685855 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:11.686454 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:11.686459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:11.688271 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
W20260923 15:12:11.688465 31512 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:11.683392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:41313 (local address 127.21.204.126:34835)
0923 15:12:11.683558 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:11.683561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:11.683725 (+ 164us) server_negotiation.cc:407] Connection header received
0923 15:12:11.683757 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:11.683758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:11.683770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:11.683835 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:11.684174 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.684699 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.685395 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.685618 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.685686 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:11.688373 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:41313: BlockingRecv error: recv got EOF from 127.21.204.65:41313 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
Sep 23 15:12:11 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:11 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:11.828678 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:11.824077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:37487)
0923 15:12:11.824264 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:11.824278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:11.824369 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:11.824529 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:11.824530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:11.824706 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:11.824753 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:11.824755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.825424 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.825425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:11.825994 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:11.825996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.826357 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.826357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:11.826415 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:11.826870 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:11.826875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:11.828570 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
W20260923 15:12:11.828718 31512 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:11.824185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:37487 (local address 127.21.204.126:34835)
0923 15:12:11.824269 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:11.824271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:11.824347 (+ 76us) server_negotiation.cc:407] Connection header received
0923 15:12:11.824427 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:11.824428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:11.824444 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:11.824506 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:11.824845 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.825339 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.826077 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.826282 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.826358 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:11.828654 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:37487: BlockingRecv error: recv got EOF from 127.21.204.66:37487 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 15:12:11 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:11 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:11.965965 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:11.960774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:53937)
0923 15:12:11.960936 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:11.960948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:11.961037 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:11.961207 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:11.961207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:11.961394 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:11.961443 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:11.961444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.962231 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.962232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:11.962759 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:11.962760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.963127 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.963128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:11.963197 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:11.963688 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:11.963693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:11.965844 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
W20260923 15:12:11.966029 31512 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:11.960975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:53937 (local address 127.21.204.126:34835)
0923 15:12:11.961042 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:11.961044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:11.961053 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:12:11.961103 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:11.961105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:11.961119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:11.961200 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:11.961532 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.962137 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.962843 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:11.963042 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:11.963126 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:11.965966 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:53937: BlockingRecv error: recv got EOF from 127.21.204.67:53937 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 15:12:12 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:12 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:12.693939 31516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:12.688931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:36719)
0923 15:12:12.689173 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:12.689188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:12.689262 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:12.689484 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:12.689485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:12.689676 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:12.689748 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:12.689750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.690517 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.690518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:12.691082 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:12.691084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.691414 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.691414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:12.691453 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:12.691943 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:12.691948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:12.693818 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":120,"threads_started":1}
W20260923 15:12:12.694087 31517 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:12.689004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:36719 (local address 127.21.204.126:34835)
0923 15:12:12.689135 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:12.689137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:12.689263 (+ 126us) server_negotiation.cc:407] Connection header received
0923 15:12:12.689376 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:12.689377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:12.689392 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:12.689455 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:12.689926 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.690435 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.691160 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.691349 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.691402 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:12.693990 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:36719: BlockingRecv error: recv got EOF from 127.21.204.65:36719 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":49,"threads_started":1}
Sep 23 15:12:12 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:12 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:12.834199 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:12.829241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:42887)
0923 15:12:12.829388 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:12.829399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:12.829450 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:12.829603 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:12.829604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:12.829801 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:12.829848 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:12.829850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.830629 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.830631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:12.831188 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:12.831190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.831574 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.831575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:12.831644 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:12.832122 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:12.832127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:12.834132 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
W20260923 15:12:12.834358 31517 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:12.829323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:42887 (local address 127.21.204.126:34835)
0923 15:12:12.829417 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:12.829419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:12.829437 (+ 18us) server_negotiation.cc:407] Connection header received
0923 15:12:12.829518 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:12.829519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:12.829535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:12.829592 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:12.830005 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.830528 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.831281 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.831497 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.831575 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:12.834235 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:42887: BlockingRecv error: recv got EOF from 127.21.204.66:42887 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 15:12:12 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:12 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:12.971184 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:12.966535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:52759)
0923 15:12:12.966668 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:12.966686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:12.966743 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:12.966949 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:12.966950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:12.967109 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:12.967158 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:12.967159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.967862 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.967863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:12.968453 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:12.968455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.968770 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.968771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:12.968815 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:12.969234 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:12.969239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:12.971075 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260923 15:12:12.971259 31517 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:12.966575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:52759 (local address 127.21.204.126:34835)
0923 15:12:12.966652 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:12.966654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:12.966751 (+ 97us) server_negotiation.cc:407] Connection header received
0923 15:12:12.966809 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:12.966810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:12.966825 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:12.966896 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:12.967226 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.967748 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.968528 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:12.968706 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:12.968756 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:12.971183 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:52759: BlockingRecv error: recv got EOF from 127.21.204.67:52759 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 15:12:13 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:13 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:13.699002 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:13.694396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:43223)
0923 15:12:13.694532 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:13.694541 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:13.694585 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:13.694815 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:13.694815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:13.694984 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:13.695034 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:13.695036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.695756 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.695757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:13.696294 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:13.696296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.696635 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.696636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:13.696676 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:13.697197 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:13.697203 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:13.698942 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":53,"threads_started":1}
W20260923 15:12:13.699122 31521 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:13.694534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:43223 (local address 127.21.204.126:34835)
0923 15:12:13.694671 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:13.694673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:13.694682 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:12:13.694705 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:13.694706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:13.694717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:13.694798 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:13.695107 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.695662 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.696372 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.696566 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.696620 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:13.699043 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:43223: BlockingRecv error: recv got EOF from 127.21.204.65:43223 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
Sep 23 15:12:13 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:13 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:13.839411 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:13.834693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:59289)
0923 15:12:13.834887 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:13.834900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:13.834987 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:13.835119 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:13.835119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:13.835317 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:13.835361 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:13.835363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.836083 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.836085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:13.836619 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:13.836621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.837007 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.837008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:13.837066 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:13.837517 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:13.837521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:13.839315 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
W20260923 15:12:13.839464 31521 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:13.834799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:59289 (local address 127.21.204.126:34835)
0923 15:12:13.834886 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:13.834888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:13.834971 (+ 83us) server_negotiation.cc:407] Connection header received
0923 15:12:13.835045 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:13.835046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:13.835060 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:13.835107 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:13.835456 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.835989 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.836716 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.836932 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.837009 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:13.839398 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:59289: BlockingRecv error: recv got EOF from 127.21.204.66:59289 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 15:12:13 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:13 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:13.976610 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:13.971930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:40163)
0923 15:12:13.972071 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:13.972082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:13.972143 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:13.972480 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:13.972481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:13.972654 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:13.972759 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:13.972762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.973464 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.973465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:13.974022 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:13.974024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.974340 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.974341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:13.974378 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:13.974800 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:13.974804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:13.976550 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
W20260923 15:12:13.976682 31521 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:13.972000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:40163 (local address 127.21.204.126:34835)
0923 15:12:13.972068 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:13.972071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:13.972295 (+ 224us) server_negotiation.cc:407] Connection header received
0923 15:12:13.972325 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:13.972326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:13.972341 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:13.972405 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:13.972894 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.973377 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.974100 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:13.974278 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:13.974330 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:13.976636 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:40163: BlockingRecv error: recv got EOF from 127.21.204.67:40163 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 15:12:14 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:14 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:14.704907 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:14.699731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:38297)
0923 15:12:14.699915 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:14.699931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:14.700039 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:14.700277 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:14.700279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:14.700485 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:14.700537 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:14.700539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.701317 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.701319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:14.701969 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:14.701971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.702418 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.702419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:14.702496 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:14.703043 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:14.703049 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:14.704847 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
W20260923 15:12:14.705155 31525 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:14.699741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:38297 (local address 127.21.204.126:34835)
0923 15:12:14.699920 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:14.699922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:14.700016 (+ 94us) server_negotiation.cc:407] Connection header received
0923 15:12:14.700125 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:14.700126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:14.700148 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:14.700264 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:14.700636 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.701223 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.702053 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.702326 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.702402 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:14.704959 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:38297: BlockingRecv error: recv got EOF from 127.21.204.65:38297 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
Sep 23 15:12:14 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:14 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:14.845301 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:14.839987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:54643)
0923 15:12:14.840144 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:14.840164 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:14.840220 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:14.840432 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:14.840433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:14.840618 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:14.840669 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:14.840670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.841471 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.841472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:14.842168 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:14.842170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.842586 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.842587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:14.842636 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:14.843200 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:14.843206 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:14.845227 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
W20260923 15:12:14.845374 31525 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:14.840165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:54643 (local address 127.21.204.126:34835)
0923 15:12:14.840244 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:14.840246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:14.840256 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:12:14.840293 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:14.840294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:14.840306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:14.840370 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:14.840793 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.841397 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.842257 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.842524 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.842589 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:14.845321 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:54643: BlockingRecv error: recv got EOF from 127.21.204.66:54643 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 15:12:14 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:14 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 15:12:14.982836 31525 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:14.977465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:40161 (local address 127.21.204.126:34835)
0923 15:12:14.977514 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:14.977516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:14.977707 (+ 191us) server_negotiation.cc:407] Connection header received
0923 15:12:14.977758 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:14.977760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:14.977866 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:14.977967 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:14.978303 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.979025 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.979899 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.980101 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.980164 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:14.982788 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:40161: BlockingRecv error: recv got EOF from 127.21.204.67:40161 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:12:14.982841 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:14.977430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:40161)
0923 15:12:14.977591 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:14.977624 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:14.977696 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:14.977978 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:14.977979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:14.978165 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:14.978211 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:14.978212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.979146 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.979147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:14.979801 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:14.979804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:14.980200 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:14.980201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:14.980258 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:14.980722 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:14.980728 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:14.982660 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
Sep 23 15:12:15 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:15 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:15.714537 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:15.705584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:42837)
0923 15:12:15.705791 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:15.705802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:15.706015 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:15.706255 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:15.706259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:15.706451 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:15.706501 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:15.706503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.707273 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.707274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:15.707842 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:15.707844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.708272 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.708273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:15.708326 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:15.712164 (+ 3838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:15.712170 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:15.714453 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
W20260923 15:12:15.714710 31529 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:15.705711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:42837 (local address 127.21.204.126:34835)
0923 15:12:15.705825 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:15.705827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:15.705857 (+ 30us) server_negotiation.cc:407] Connection header received
0923 15:12:15.706113 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:15.706114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:15.706127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:15.706194 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:15.706617 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.707160 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.707951 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.708153 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.708215 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:15.714604 (+ 6389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:42837: BlockingRecv error: recv got EOF from 127.21.204.65:42837 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
Sep 23 15:12:15 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:15 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:15.850965 31530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:15.845886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:60659)
0923 15:12:15.846030 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:15.846042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:15.846111 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:15.846300 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:15.846301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:15.846461 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:15.846509 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:15.846511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.847172 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.847173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:15.847724 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:15.847726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.848145 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.848146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:15.848200 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:15.848649 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:15.848654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:15.850868 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
W20260923 15:12:15.851009 31529 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:15.845998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:60659 (local address 127.21.204.126:34835)
0923 15:12:15.846071 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:15.846073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:15.846098 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:12:15.846178 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:15.846179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:15.846190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:15.846246 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:15.846583 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.847095 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.847814 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.848036 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.848093 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:15.850947 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:60659: BlockingRecv error: recv got EOF from 127.21.204.66:60659 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 15:12:15 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:15 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.67@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:15.988106 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:15.983381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.67:44987)
0923 15:12:15.983567 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:15.983585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:15.983682 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:15.983856 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:15.983857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:15.984042 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:15.984090 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:15.984092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.984771 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.984772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:15.985310 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:15.985313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.985690 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.985691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:15.985748 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:15.986279 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:15.986284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:15.988010 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
W20260923 15:12:15.988166 31529 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:15.983485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:44987 (local address 127.21.204.126:34835)
0923 15:12:15.983571 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:15.983574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:15.983662 (+ 88us) server_negotiation.cc:407] Connection header received
0923 15:12:15.983757 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:15.983758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:15.983773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:15.983846 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:15.984173 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.984686 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.985402 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:15.985602 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:15.985694 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:15.988090 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.67:44987: BlockingRecv error: recv got EOF from 127.21.204.67:44987 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 15:12:16 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:16 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.65@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:16.720760 31532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:16.715177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.65:56779)
0923 15:12:16.715339 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:16.715349 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:16.715400 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:16.715714 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:16.715716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:16.715905 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:16.715958 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:16.715960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.716728 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.716729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:16.717414 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:16.717417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.717809 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.717810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:16.717954 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:16.718509 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:16.718514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:16.720696 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
W20260923 15:12:16.720899 31533 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:16.715281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:56779 (local address 127.21.204.126:34835)
0923 15:12:16.715428 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:16.715430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:16.715438 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:12:16.715462 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:16.715462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:16.715472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:16.715538 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:16.716064 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.716596 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.717509 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.717713 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.717777 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:16.720776 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.65:56779: BlockingRecv error: recv got EOF from 127.21.204.65:56779 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
Sep 23 15:12:16 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 15:12:16 dist-test-slave-t5lt krb5kdc[30472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.204.66@KRBTEST.COM for oryx/127.21.204.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 15:12:16.856498 31534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:16.851520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34835 (local address 127.21.204.66:39587)
0923 15:12:16.851732 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:16.851745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:16.851855 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:16.852039 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:16.852040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:16.852219 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:16.852267 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:16.852269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.852958 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.852959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:16.853496 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:16.853498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.853861 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.853862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:16.853967 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:16.854496 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:16.854504 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:16.856438 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.204.126:34835: Server oryx/127.21.204.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
W20260923 15:12:16.856628 31533 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:16.851609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:39587 (local address 127.21.204.126:34835)
0923 15:12:16.851712 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:16.851715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:16.851836 (+ 121us) server_negotiation.cc:407] Connection header received
0923 15:12:16.851937 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:16.851938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:16.851953 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:16.852028 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:16.852352 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.852870 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.853585 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:16.853786 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:16.853863 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:16.856546 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.204.66:39587: BlockingRecv error: recv got EOF from 127.21.204.66:39587 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 15:12:16.893198 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 31055
I20260923 15:12:16.898208 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 31208
I20260923 15:12:16.902554 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 31342
I20260923 15:12:16.906796 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 30985
2026-09-23T15:12:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17528 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T15:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:12:16Z Disabled control of system clock
I20260923 15:12:16.934108 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:36413
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:36413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 15:12:17.013482 31538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.013697 31538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.013718 31538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.014952 31538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:12:17.014995 31538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.015008 31538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.015021 31538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:12:17.015033 31538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:12:17.016326 31538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:36413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:36413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.016525 31538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.016770 31538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.021883 31544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.021896 31546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.021992 31538 server_base.cc:1034] running on GCE node
W20260923 15:12:17.022107 31543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.022409 31538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.022713 31538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.023880 31538 hybrid_clock.cc:648] HybridClock initialized: now 1790176337023871 us; error 32 us; skew 500 ppm
I20260923 15:12:17.025143 31538 webserver.cc:460] Webserver started at http://127.21.204.126:39825/ using document root <none> and password file <none>
I20260923 15:12:17.025405 31538 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.025481 31538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.025635 31538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.026751 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "50c3802fd77d4686bca510a4a38355c6"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "46544ae0d05a1d6de8d916fc228f17b1"
server_key_iv: "1a7d72a60eb8d8534d47c78a8fed7254"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.027139 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "50c3802fd77d4686bca510a4a38355c6"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "46544ae0d05a1d6de8d916fc228f17b1"
server_key_iv: "1a7d72a60eb8d8534d47c78a8fed7254"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.028472 31538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:12:17.029297 31552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.029524 31538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:17.029624 31538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "50c3802fd77d4686bca510a4a38355c6"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "46544ae0d05a1d6de8d916fc228f17b1"
server_key_iv: "1a7d72a60eb8d8534d47c78a8fed7254"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.029707 31538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.045322 31538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.045651 31538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.045812 31538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.049793 31538 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:36413
I20260923 15:12:17.049844 31604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:36413 every 8 connection(s)
I20260923 15:12:17.050221 31538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:12:17.050940 31605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:12:17.053678 31605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6: Bootstrap starting.
I20260923 15:12:17.054343 31605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6: Neither blocks nor log segments found. Creating new log.
I20260923 15:12:17.054615 31605 log.cc:824] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:17.055328 31605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6: No bootstrap required, opened a new log
I20260923 15:12:17.056918 31605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c3802fd77d4686bca510a4a38355c6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 36413 } }
I20260923 15:12:17.057054 31605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:12:17.057076 31605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c3802fd77d4686bca510a4a38355c6, State: Initialized, Role: FOLLOWER
I20260923 15:12:17.057227 31605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c3802fd77d4686bca510a4a38355c6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 36413 } }
I20260923 15:12:17.057315 31605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:12:17.057337 31605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:12:17.057366 31605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:12:17.057857 31605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c3802fd77d4686bca510a4a38355c6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 36413 } }
I20260923 15:12:17.057960 31605 leader_election.cc:302] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50c3802fd77d4686bca510a4a38355c6; no voters:
I20260923 15:12:17.058079 31605 leader_election.cc:288] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:12:17.058159 31608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:12:17.058336 31608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [term 1 LEADER]: Becoming Leader. State: Replica: 50c3802fd77d4686bca510a4a38355c6, State: Running, Role: LEADER
I20260923 15:12:17.058398 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 31538
I20260923 15:12:17.058434 31608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c3802fd77d4686bca510a4a38355c6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 36413 } }
I20260923 15:12:17.058494 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:12:17.058745 22321 external_mini_cluster.cc:1428] Setting key 6c7e60cafa703747c2f33cd608a53d9b
I20260923 15:12:17.058781 31609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50c3802fd77d4686bca510a4a38355c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c3802fd77d4686bca510a4a38355c6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 36413 } } }
I20260923 15:12:17.058852 31609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:17.058913 31605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:12:17.059165 31610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c3802fd77d4686bca510a4a38355c6. Latest consensus state: current_term: 1 leader_uuid: "50c3802fd77d4686bca510a4a38355c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c3802fd77d4686bca510a4a38355c6" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 36413 } } }
I20260923 15:12:17.059237 31612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:12:17.060045 31612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:12:17.060421 31610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:17.062889 31612 catalog_manager.cc:1347] Generated new cluster ID: 0a3264b07948454a8a8431b5caf40a4e
I20260923 15:12:17.062973 31612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:12:17.065347 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.059451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.21.204.126:36413)
0923 15:12:17.059671 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.059695 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.059730 (+ 35us) server_negotiation.cc:407] Connection header received
0923 15:12:17.059858 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.059867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.059940 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.060314 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.060424 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.061824 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.062857 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.063495 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.063623 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:17.064407 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:17.064411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:17.064411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:17.064620 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:17.064664 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:17.064691 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:17.064760 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:17.064952 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 15:12:17.065135 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260923 15:12:17.076207 31612 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:12:17.076695 31612 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:12:17.084048 31612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50c3802fd77d4686bca510a4a38355c6: Generated new TSK 0
I20260923 15:12:17.084257 31612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:12:17.089334 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:36413
--builtin_ntp_servers=127.21.204.84:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 15:12:17.170428 31629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.170655 31629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.170696 31629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.171859 31629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.171919 31629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.171963 31629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:12:17.173309 31629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:36413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.173563 31629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.173950 31629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.179138 31634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.179266 31629 server_base.cc:1034] running on GCE node
W20260923 15:12:17.179126 31637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.179131 31635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.179605 31629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.179859 31629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.181025 31629 hybrid_clock.cc:648] HybridClock initialized: now 1790176337181005 us; error 37 us; skew 500 ppm
I20260923 15:12:17.182305 31629 webserver.cc:460] Webserver started at http://127.21.204.65:39333/ using document root <none> and password file <none>
I20260923 15:12:17.182548 31629 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.182617 31629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.182750 31629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.183807 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "23c6e86c2dc44295bbc00dc262bba7f7"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "1c6e83a2a2b5836dd9391ecf3cabb744"
server_key_iv: "0c85b2d3cf092118a7cf43602055c9c7"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.184311 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "23c6e86c2dc44295bbc00dc262bba7f7"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "1c6e83a2a2b5836dd9391ecf3cabb744"
server_key_iv: "0c85b2d3cf092118a7cf43602055c9c7"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.185670 31629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:12:17.186570 31643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.186902 31629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:17.187004 31629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "23c6e86c2dc44295bbc00dc262bba7f7"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "1c6e83a2a2b5836dd9391ecf3cabb744"
server_key_iv: "0c85b2d3cf092118a7cf43602055c9c7"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.187086 31629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.199473 31629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.199760 31629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.199908 31629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.200126 31629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:17.200443 31629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:17.200500 31629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.200539 31629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:17.200567 31629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.206305 31629 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:46623
I20260923 15:12:17.206406 31756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:46623 every 8 connection(s)
I20260923 15:12:17.206707 31629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:12:17.212167 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.207477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:44369 (local address 127.21.204.126:36413)
0923 15:12:17.207594 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.207597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.207804 (+ 207us) server_negotiation.cc:407] Connection header received
0923 15:12:17.208403 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.208404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.208420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.208474 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.209130 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.209671 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.210738 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.210940 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.211019 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:17.211882 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:17.211883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:17.211883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:17.211889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:17.211904 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:17.211908 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:17.211980 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:17.212059 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 15:12:17.212088 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 15:12:17.212386 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.207401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:36413 (local address 127.21.204.65:44369)
0923 15:12:17.207660 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:17.207706 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:17.208275 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:17.208542 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:17.208546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:17.208556 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:17.209036 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:17.209040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.209785 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.209786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:17.210632 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:17.210634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.211027 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.211028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:17.211168 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:17.211727 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:12:17.211727 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:17.211755 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:12:17.211776 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:17.211979 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:17.211986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:17.211988 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:12:17.212061 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 15:12:17.212159 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260923 15:12:17.212914 31757 heartbeater.cc:347] Connected to a master server at 127.21.204.126:36413
I20260923 15:12:17.212991 31757 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:17.213189 31757 heartbeater.cc:511] Master 127.21.204.126:36413 requested a full tablet report, sending...
I20260923 15:12:17.213737 31569 ts_manager.cc:194] Registered new tserver with Master: 23c6e86c2dc44295bbc00dc262bba7f7 (127.21.204.65:46623)
I20260923 15:12:17.214385 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 31629
I20260923 15:12:17.214390 31569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.65:44369
I20260923 15:12:17.214522 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:12:17.214823 22321 external_mini_cluster.cc:1428] Setting key 3644a988889fa947f31334e516819d6e
I20260923 15:12:17.215673 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:36413
--builtin_ntp_servers=127.21.204.84:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 15:12:17.298859 31760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.299067 31760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.299088 31760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.300226 31760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.300264 31760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.300289 31760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:12:17.301604 31760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:36413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.301852 31760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.302201 31760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.307363 31768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.307413 31766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.307549 31760 server_base.cc:1034] running on GCE node
W20260923 15:12:17.307440 31765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.307860 31760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.308097 31760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.309267 31760 hybrid_clock.cc:648] HybridClock initialized: now 1790176337309250 us; error 35 us; skew 500 ppm
I20260923 15:12:17.310616 31760 webserver.cc:460] Webserver started at http://127.21.204.66:39891/ using document root <none> and password file <none>
I20260923 15:12:17.310876 31760 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.310925 31760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.311056 31760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.312065 31760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "4f6aba38ab874eebb6cc93c19349fabb"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "d55455539d8508763e53dba2d1d63f48"
server_key_iv: "69b59cc1bf388c002b372d536eb03477"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.312407 31760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "4f6aba38ab874eebb6cc93c19349fabb"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "d55455539d8508763e53dba2d1d63f48"
server_key_iv: "69b59cc1bf388c002b372d536eb03477"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.313680 31760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:12:17.314538 31774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.314738 31760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:17.314857 31760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "4f6aba38ab874eebb6cc93c19349fabb"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "d55455539d8508763e53dba2d1d63f48"
server_key_iv: "69b59cc1bf388c002b372d536eb03477"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.314932 31760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.336621 31760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.336947 31760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.337102 31760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.337337 31760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:17.337670 31760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:17.337733 31760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.337769 31760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:17.337796 31760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.343776 31760 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:43187
I20260923 15:12:17.343828 31887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:43187 every 8 connection(s)
I20260923 15:12:17.344146 31760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:12:17.349409 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.344916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:37599 (local address 127.21.204.126:36413)
0923 15:12:17.344988 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.344990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.345242 (+ 252us) server_negotiation.cc:407] Connection header received
0923 15:12:17.345712 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.345713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.345724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.345779 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.346448 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.346969 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.347986 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.348167 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.348229 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:17.349101 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:17.349103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:17.349103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:17.349107 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:17.349122 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:17.349126 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:17.349187 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:17.349311 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 15:12:17.349336 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 15:12:17.349598 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.344891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:36413 (local address 127.21.204.66:37599)
0923 15:12:17.345101 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:17.345142 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:17.345628 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:17.345838 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:17.345841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:17.345849 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:17.346366 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:17.346370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.347080 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.347080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:17.347895 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:17.347898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.348278 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.348279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:17.348394 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:17.348960 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:12:17.348960 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:17.348988 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:12:17.349011 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:17.349202 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:17.349219 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:17.349222 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:12:17.349277 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 15:12:17.349359 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260923 15:12:17.350274 31888 heartbeater.cc:347] Connected to a master server at 127.21.204.126:36413
I20260923 15:12:17.350486 31888 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:17.350668 31888 heartbeater.cc:511] Master 127.21.204.126:36413 requested a full tablet report, sending...
I20260923 15:12:17.350656 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 31760
I20260923 15:12:17.350798 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:12:17.351056 22321 external_mini_cluster.cc:1428] Setting key ff7e7f79b7af225c1479f188fbfc1562
I20260923 15:12:17.351145 31569 ts_manager.cc:194] Registered new tserver with Master: 4f6aba38ab874eebb6cc93c19349fabb (127.21.204.66:43187)
I20260923 15:12:17.351575 31569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.66:37599
I20260923 15:12:17.352061 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:36413
--builtin_ntp_servers=127.21.204.84:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 15:12:17.432545 31891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.432747 31891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.432803 31891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.433995 31891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.434032 31891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.434057 31891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:12:17.435282 31891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:36413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.435479 31891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.435709 31891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.440640 31899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.440686 31896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.440694 31897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.440991 31891 server_base.cc:1034] running on GCE node
I20260923 15:12:17.441221 31891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.441473 31891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.442615 31891 hybrid_clock.cc:648] HybridClock initialized: now 1790176337442618 us; error 44 us; skew 500 ppm
I20260923 15:12:17.443753 31891 webserver.cc:460] Webserver started at http://127.21.204.67:43511/ using document root <none> and password file <none>
I20260923 15:12:17.443996 31891 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.444041 31891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.444182 31891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.445219 31891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "48f407a0a6944b4682746c342147940b"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "26445773d6fff18c84b73afd836f195a"
server_key_iv: "1dbac87d143f47743df6a7987b14bf83"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.445576 31891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "48f407a0a6944b4682746c342147940b"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "26445773d6fff18c84b73afd836f195a"
server_key_iv: "1dbac87d143f47743df6a7987b14bf83"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.446933 31891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:12:17.447778 31905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.448019 31891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:17.448117 31891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "48f407a0a6944b4682746c342147940b"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "26445773d6fff18c84b73afd836f195a"
server_key_iv: "1dbac87d143f47743df6a7987b14bf83"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.448192 31891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.457814 31891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.458108 31891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.458256 31891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.458473 31891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:17.458803 31891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:17.458838 31891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.458858 31891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:17.458890 31891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.464668 31891 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:34167
I20260923 15:12:17.464707 32018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:34167 every 8 connection(s)
I20260923 15:12:17.465013 31891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:12:17.466936 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 31891
I20260923 15:12:17.467052 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:12:17.467284 22321 external_mini_cluster.cc:1428] Setting key 0c6e7d59fcd5dba6ae9d10d7a9453370
I20260923 15:12:17.470319 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.465672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:37869 (local address 127.21.204.126:36413)
0923 15:12:17.465739 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.465751 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.465978 (+ 227us) server_negotiation.cc:407] Connection header received
0923 15:12:17.466444 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.466445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.466457 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.466509 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.467172 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.467941 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.469049 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.469228 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.469285 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:17.470123 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:17.470124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:17.470124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:17.470128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:17.470139 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:17.470142 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:17.470182 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:17.470240 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 15:12:17.470263 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:12:17.470544 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.465662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:36413 (local address 127.21.204.67:37869)
0923 15:12:17.465852 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:17.465897 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:17.466351 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:17.466560 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:17.466562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:17.466570 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:17.467084 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:17.467090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.468054 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.468055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:17.468970 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:17.468973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:17.469305 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:17.469306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:17.469423 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:17.470000 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:12:17.470001 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:17.470027 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:12:17.470049 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:17.470183 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:17.470185 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:17.470187 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:12:17.470232 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:12:17.470303 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
I20260923 15:12:17.471009 32019 heartbeater.cc:347] Connected to a master server at 127.21.204.126:36413
I20260923 15:12:17.471107 32019 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:17.471282 32019 heartbeater.cc:511] Master 127.21.204.126:36413 requested a full tablet report, sending...
I20260923 15:12:17.471642 31568 ts_manager.cc:194] Registered new tserver with Master: 48f407a0a6944b4682746c342147940b (127.21.204.67:34167)
I20260923 15:12:17.471979 31568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.67:37869
I20260923 15:12:17.479103 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 15:12:17.480973 31618 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:17.480442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.21.204.126:36413)
0923 15:12:17.480522 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.480524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.480622 (+ 98us) server_negotiation.cc:407] Connection header received
0923 15:12:17.480696 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.480697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.480709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.480777 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.480908 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52612: BlockingRecv error: recv got EOF from 127.0.0.1:52612 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 15:12:17.481068 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 31629
I20260923 15:12:17.485733 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 31760
I20260923 15:12:17.489764 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 31891
I20260923 15:12:17.493737 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 31538
2026-09-23T15:12:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (586 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T15:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:12:17Z Disabled control of system clock
I20260923 15:12:17.520673 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:44353
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 15:12:17.601595 32030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.601850 32030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.601888 32030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.603083 32030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:12:17.603170 32030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.603199 32030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.603235 32030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:12:17.603263 32030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:12:17.604635 32030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:44353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.604926 32030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.605224 32030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.610170 32035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.610324 32036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.610388 32030 server_base.cc:1034] running on GCE node
W20260923 15:12:17.610507 32038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.610834 32030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.611123 32030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.612298 32030 hybrid_clock.cc:648] HybridClock initialized: now 1790176337612264 us; error 53 us; skew 500 ppm
I20260923 15:12:17.613487 32030 webserver.cc:460] Webserver started at http://127.21.204.126:36411/ using document root <none> and password file <none>
I20260923 15:12:17.613752 32030 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.613826 32030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.614007 32030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.615100 32030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "5bdce81c75d54d75b032e9f7baf27a39"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "ae0128ab654dda794d0718237a9dd90c"
server_key_iv: "02372068bda0abee1a8a7209e209c58c"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.615473 32030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "5bdce81c75d54d75b032e9f7baf27a39"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "ae0128ab654dda794d0718237a9dd90c"
server_key_iv: "02372068bda0abee1a8a7209e209c58c"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.616876 32030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:12:17.617668 32044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.617908 32030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:12:17.618016 32030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "5bdce81c75d54d75b032e9f7baf27a39"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "ae0128ab654dda794d0718237a9dd90c"
server_key_iv: "02372068bda0abee1a8a7209e209c58c"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.618104 32030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.621351 32030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.621613 32030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.621757 32030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.626008 32030 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:44353
I20260923 15:12:17.626060 32096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:44353 every 8 connection(s)
I20260923 15:12:17.626508 32030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:12:17.627174 32097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:12:17.629971 32097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39: Bootstrap starting.
I20260923 15:12:17.630497 32097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39: Neither blocks nor log segments found. Creating new log.
I20260923 15:12:17.630744 32097 log.cc:824] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:17.631388 32097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39: No bootstrap required, opened a new log
I20260923 15:12:17.632959 32097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdce81c75d54d75b032e9f7baf27a39" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44353 } }
I20260923 15:12:17.633088 32097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:12:17.633109 32097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdce81c75d54d75b032e9f7baf27a39, State: Initialized, Role: FOLLOWER
I20260923 15:12:17.633250 32097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdce81c75d54d75b032e9f7baf27a39" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44353 } }
I20260923 15:12:17.633337 32097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:12:17.633361 32097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:12:17.633388 32097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:12:17.633867 32097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdce81c75d54d75b032e9f7baf27a39" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44353 } }
I20260923 15:12:17.634018 32097 leader_election.cc:302] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bdce81c75d54d75b032e9f7baf27a39; no voters:
I20260923 15:12:17.634143 32097 leader_election.cc:288] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:12:17.634231 32100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:12:17.634406 32100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [term 1 LEADER]: Becoming Leader. State: Replica: 5bdce81c75d54d75b032e9f7baf27a39, State: Running, Role: LEADER
I20260923 15:12:17.634424 32097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:12:17.634492 32100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdce81c75d54d75b032e9f7baf27a39" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44353 } }
I20260923 15:12:17.634920 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 32030
I20260923 15:12:17.635040 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:12:17.635157 32102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bdce81c75d54d75b032e9f7baf27a39. Latest consensus state: current_term: 1 leader_uuid: "5bdce81c75d54d75b032e9f7baf27a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdce81c75d54d75b032e9f7baf27a39" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44353 } } }
I20260923 15:12:17.635245 32102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:17.635272 22321 external_mini_cluster.cc:1428] Setting key 842b02814f67f053672d320950b7f326
I20260923 15:12:17.635401 32101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bdce81c75d54d75b032e9f7baf27a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdce81c75d54d75b032e9f7baf27a39" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44353 } } }
I20260923 15:12:17.635470 32101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:17.635612 32109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:12:17.636258 32109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:12:17.637931 32109 catalog_manager.cc:1347] Generated new cluster ID: e1176f1fb90f44d9a2be97d14c1c4d00
I20260923 15:12:17.637995 32109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:12:17.638378 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.635949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.21.204.126:44353)
0923 15:12:17.636490 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.636499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.636527 (+ 28us) server_negotiation.cc:407] Connection header received
0923 15:12:17.636797 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.636807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.636857 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.637022 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.637743 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:17.637746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:17.637747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:17.637751 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:17.637823 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:17.637838 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:17.638131 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:17.638163 (+ 32us) server_negotiation.cc:299] Negotiation successful
0923 15:12:17.638220 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":80,"threads_started":1}
I20260923 15:12:17.643992 32109 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:12:17.644147 32109 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:12:17.651859 32109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bdce81c75d54d75b032e9f7baf27a39: Generated new TSK 0
I20260923 15:12:17.652025 32109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:12:17.659127 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44353
--builtin_ntp_servers=127.21.204.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 15:12:17.741698 32121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.741963 32121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.742002 32121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.743165 32121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.743229 32121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.743273 32121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:12:17.744616 32121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.744885 32121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.745167 32121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.751026 32127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.751134 32129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.751024 32126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.751245 32121 server_base.cc:1034] running on GCE node
I20260923 15:12:17.751467 32121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.751742 32121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.752905 32121 hybrid_clock.cc:648] HybridClock initialized: now 1790176337752883 us; error 36 us; skew 500 ppm
I20260923 15:12:17.754388 32121 webserver.cc:460] Webserver started at http://127.21.204.65:43629/ using document root <none> and password file <none>
I20260923 15:12:17.754635 32121 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.754679 32121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.754803 32121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.755861 32121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "6257b07aa45c4c178add9939540a73df"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "5f65739fb9c575efdcff09a07e930a74"
server_key_iv: "90254bb4efa9a071752e96147d92b238"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.756213 32121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "6257b07aa45c4c178add9939540a73df"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "5f65739fb9c575efdcff09a07e930a74"
server_key_iv: "90254bb4efa9a071752e96147d92b238"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.757531 32121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:12:17.758385 32135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.758538 32121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.758623 32121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "6257b07aa45c4c178add9939540a73df"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "5f65739fb9c575efdcff09a07e930a74"
server_key_iv: "90254bb4efa9a071752e96147d92b238"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.758692 32121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.761915 32121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.762193 32121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.762323 32121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.762567 32121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:17.762897 32121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:17.762928 32121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.762976 32121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:17.763001 32121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.768888 32121 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:46189
I20260923 15:12:17.768923 32248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:46189 every 8 connection(s)
I20260923 15:12:17.769248 32121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:12:17.772159 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.770080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:38381 (local address 127.21.204.126:44353)
0923 15:12:17.770156 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.770158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.770398 (+ 240us) server_negotiation.cc:407] Connection header received
0923 15:12:17.770843 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.770844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.770850 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.770917 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.771981 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:17.771983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:17.771983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:17.771987 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:17.772001 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:17.772005 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:17.772053 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:17.772096 (+ 43us) server_negotiation.cc:299] Negotiation successful
0923 15:12:17.772108 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 15:12:17.772392 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.770064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:44353 (local address 127.21.204.65:38381)
0923 15:12:17.770275 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:17.770308 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:17.770740 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:17.770966 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:17.770969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:17.770977 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:17.771842 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:12:17.771843 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:17.771873 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:12:17.771899 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:17.772050 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:17.772052 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:17.772055 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:12:17.772104 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 15:12:17.772148 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
I20260923 15:12:17.772907 32249 heartbeater.cc:347] Connected to a master server at 127.21.204.126:44353
I20260923 15:12:17.772995 32249 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:17.773155 32249 heartbeater.cc:511] Master 127.21.204.126:44353 requested a full tablet report, sending...
I20260923 15:12:17.773644 32061 ts_manager.cc:194] Registered new tserver with Master: 6257b07aa45c4c178add9939540a73df (127.21.204.65:46189)
I20260923 15:12:17.774171 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 32121
I20260923 15:12:17.774253 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:12:17.774473 22321 external_mini_cluster.cc:1428] Setting key 754f59b593ef5fc5f6d5238a54b9205e
I20260923 15:12:17.775312 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44353
--builtin_ntp_servers=127.21.204.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 15:12:17.858958 32252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.859179 32252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.859203 32252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.860525 32252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.860569 32252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.860598 32252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:12:17.862313 32252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.862566 32252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.862891 32252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.868319 32258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.868395 32257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.868443 32260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.868489 32252 server_base.cc:1034] running on GCE node
I20260923 15:12:17.868818 32252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.869061 32252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.870219 32252 hybrid_clock.cc:648] HybridClock initialized: now 1790176337870207 us; error 43 us; skew 500 ppm
I20260923 15:12:17.871569 32252 webserver.cc:460] Webserver started at http://127.21.204.66:40377/ using document root <none> and password file <none>
I20260923 15:12:17.871824 32252 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.871899 32252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.872053 32252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.873099 32252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "cfe83dce28634c36a351bc95275120d0"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "e36aefae9683cb7538840679cc4ffab9"
server_key_iv: "f959279ed6ea7b4b56051163a21a66c7"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.873457 32252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "cfe83dce28634c36a351bc95275120d0"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "e36aefae9683cb7538840679cc4ffab9"
server_key_iv: "f959279ed6ea7b4b56051163a21a66c7"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.874837 32252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:12:17.875579 32266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.875809 32252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:17.875911 32252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "cfe83dce28634c36a351bc95275120d0"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "e36aefae9683cb7538840679cc4ffab9"
server_key_iv: "f959279ed6ea7b4b56051163a21a66c7"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.875989 32252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.879317 32252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.879590 32252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.879715 32252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.879880 32252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:17.880172 32252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:17.880224 32252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.880263 32252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:17.880294 32252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.885814 32252 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:33489
I20260923 15:12:17.885848 32379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:33489 every 8 connection(s)
I20260923 15:12:17.886183 32252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:12:17.888938 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.886922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:57775 (local address 127.21.204.126:44353)
0923 15:12:17.886994 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:17.886997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:17.887208 (+ 211us) server_negotiation.cc:407] Connection header received
0923 15:12:17.887626 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:17.887627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:17.887631 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:17.887689 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:17.888718 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:17.888719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:17.888719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:17.888723 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:17.888737 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:17.888741 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:17.888811 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:17.888874 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 15:12:17.888886 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 15:12:17.889114 32382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:17.886896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:44353 (local address 127.21.204.66:57775)
0923 15:12:17.887082 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:17.887121 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:17.887547 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:17.887738 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:17.887741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:17.887749 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:17.888585 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:12:17.888586 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:17.888612 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:12:17.888637 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:17.888810 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:17.888812 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:17.888814 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:12:17.888879 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 15:12:17.888924 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260923 15:12:17.889609 32380 heartbeater.cc:347] Connected to a master server at 127.21.204.126:44353
I20260923 15:12:17.889714 32380 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:17.889907 32380 heartbeater.cc:511] Master 127.21.204.126:44353 requested a full tablet report, sending...
I20260923 15:12:17.889863 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 32252
I20260923 15:12:17.889997 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:12:17.890195 22321 external_mini_cluster.cc:1428] Setting key c940c584bca9e15f12ae2c53e665d093
I20260923 15:12:17.890313 32061 ts_manager.cc:194] Registered new tserver with Master: cfe83dce28634c36a351bc95275120d0 (127.21.204.66:33489)
I20260923 15:12:17.891101 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44353
--builtin_ntp_servers=127.21.204.84:33105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 15:12:17.974094 32383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:17.974318 32383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:17.974356 32383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:17.975503 32383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:17.975564 32383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:17.975607 32383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:12:17.976922 32383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:33105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:17.977180 32383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:17.977465 32383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:17.982528 32388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:17.982561 32389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.982774 32383 server_base.cc:1034] running on GCE node
W20260923 15:12:17.982535 32391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:17.983086 32383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:17.983302 32383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:17.984479 32383 hybrid_clock.cc:648] HybridClock initialized: now 1790176337984466 us; error 37 us; skew 500 ppm
I20260923 15:12:17.985818 32383 webserver.cc:460] Webserver started at http://127.21.204.67:45211/ using document root <none> and password file <none>
I20260923 15:12:17.986094 32383 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:17.986143 32383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:17.986232 32383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:17.987221 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "2e9098493d9a48619da18357c10a4ce1"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "ad1c8456a71b76692883d30df926f869"
server_key_iv: "577b362ad96dbdb0798985861ad893b6"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.987517 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "2e9098493d9a48619da18357c10a4ce1"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "ad1c8456a71b76692883d30df926f869"
server_key_iv: "577b362ad96dbdb0798985861ad893b6"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.988776 32383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:12:17.989475 32397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.989665 32383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:12:17.989769 32383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "2e9098493d9a48619da18357c10a4ce1"
format_stamp: "Formatted at 2026-09-23 15:12:17 on dist-test-slave-t5lt"
server_key: "ad1c8456a71b76692883d30df926f869"
server_key_iv: "577b362ad96dbdb0798985861ad893b6"
server_key_version: "encryptionkey@0"
I20260923 15:12:17.989842 32383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:17.993120 32383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:17.993371 32383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:17.993512 32383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:17.993727 32383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:17.994074 32383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:17.994128 32383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.994168 32383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:17.994194 32383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:17.999859 32383 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:40465
I20260923 15:12:17.999890 32510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:40465 every 8 connection(s)
I20260923 15:12:18.000234 32383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:12:18.003304 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:18.001050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:38077 (local address 127.21.204.126:44353)
0923 15:12:18.001129 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:18.001131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:18.001363 (+ 232us) server_negotiation.cc:407] Connection header received
0923 15:12:18.001786 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:18.001787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:18.001793 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:18.001865 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:18.003073 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:18.003075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:18.003075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:12:18.003080 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:18.003094 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:12:18.003098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:18.003163 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:18.003237 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 15:12:18.003250 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 15:12:18.003463 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:18.001018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:44353 (local address 127.21.204.67:38077)
0923 15:12:18.001206 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:18.001246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:18.001687 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:18.002010 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:18.002014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:18.002022 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:18.002918 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:12:18.002918 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:18.002946 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:12:18.002972 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:18.003165 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:18.003170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:18.003172 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:12:18.003229 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 15:12:18.003277 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260923 15:12:18.003942 32511 heartbeater.cc:347] Connected to a master server at 127.21.204.126:44353
I20260923 15:12:18.004027 32511 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:18.004199 32511 heartbeater.cc:511] Master 127.21.204.126:44353 requested a full tablet report, sending...
I20260923 15:12:18.004570 32061 ts_manager.cc:194] Registered new tserver with Master: 2e9098493d9a48619da18357c10a4ce1 (127.21.204.67:40465)
I20260923 15:12:18.005410 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 32383
I20260923 15:12:18.005489 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:12:18.005759 22321 external_mini_cluster.cc:1428] Setting key 8736ae7c8d315c4302a9f927d30cd243
I20260923 15:12:18.006654 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 15:12:18.008447 32119 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:12:18.007956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.21.204.126:44353)
0923 15:12:18.008017 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:18.008019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:18.008152 (+ 133us) server_negotiation.cc:407] Connection header received
0923 15:12:18.008194 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:18.008195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:18.008199 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:18.008245 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:18.008387 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35450: BlockingRecv error: recv got EOF from 127.0.0.1:35450 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:12:18.008517 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 32121
I20260923 15:12:18.014601 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 32252
I20260923 15:12:18.019191 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 32383
I20260923 15:12:18.023840 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 32030
2026-09-23T15:12:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (530 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:12:18 dist-test-slave-t5lt krb5kdc[32521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:12:18 dist-test-slave-t5lt krb5kdc[32521](info): set up 2 sockets
Sep 23 15:12:18 dist-test-slave-t5lt krb5kdc[32521](info): commencing operation
krb5kdc: starting...
W20260923 15:12:19.072221 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:12:19.193531 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:44249
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:19.274299 32534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:19.274514 32534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:19.274537 32534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:19.275620 32534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:12:19.275655 32534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:19.275669 32534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:19.275683 32534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:12:19.275694 32534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:12:19.277096 32534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:44249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:44249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:19.277283 32534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:19.277465 32534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:19.282438 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:19.282497 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:19.282433 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:19.282769 32534 server_base.cc:1034] running on GCE node
I20260923 15:12:19.283047 32534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:19.283339 32534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:19.284540 32534 hybrid_clock.cc:648] HybridClock initialized: now 1790176339284520 us; error 47 us; skew 500 ppm
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:19.287215 32534 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:12:19.287678 32534 webserver.cc:460] Webserver started at http://127.21.204.126:32887/ using document root <none> and password file <none>
I20260923 15:12:19.287935 32534 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:19.287981 32534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:19.288069 32534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:19.289059 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "02b49cedffa444da9e85485fbaef06a2"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "b60eba2345e06f912584a8a27615aa48"
server_key_iv: "9ce9d64cfbb7b793f256840c183687f6"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.289343 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "02b49cedffa444da9e85485fbaef06a2"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "b60eba2345e06f912584a8a27615aa48"
server_key_iv: "9ce9d64cfbb7b793f256840c183687f6"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.290673 32534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:12:19.291339 32549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.291522 32534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:12:19.291589 32534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "02b49cedffa444da9e85485fbaef06a2"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "b60eba2345e06f912584a8a27615aa48"
server_key_iv: "9ce9d64cfbb7b793f256840c183687f6"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.291638 32534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:19.298805 32534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:19.300798 32534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:19.300928 32534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:19.305094 32534 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:44249
I20260923 15:12:19.305124 32601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:44249 every 8 connection(s)
I20260923 15:12:19.305485 32534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:12:19.306489 32602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:12:19.308145 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 32534
I20260923 15:12:19.308231 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:12:19.308466 22321 external_mini_cluster.cc:1428] Setting key 9c2490096fca45bb0fae82885c3f8062
I20260923 15:12:19.308902 32602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2: Bootstrap starting.
I20260923 15:12:19.309581 32602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2: Neither blocks nor log segments found. Creating new log.
I20260923 15:12:19.309917 32602 log.cc:824] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:19.310592 32602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2: No bootstrap required, opened a new log
I20260923 15:12:19.312325 32602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b49cedffa444da9e85485fbaef06a2" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44249 } }
I20260923 15:12:19.312485 32602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:12:19.312532 32602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b49cedffa444da9e85485fbaef06a2, State: Initialized, Role: FOLLOWER
I20260923 15:12:19.312659 32602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b49cedffa444da9e85485fbaef06a2" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44249 } }
I20260923 15:12:19.312767 32602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:12:19.312812 32602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:12:19.312867 32602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:12:19.313494 32602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b49cedffa444da9e85485fbaef06a2" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44249 } }
I20260923 15:12:19.313583 32602 leader_election.cc:302] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b49cedffa444da9e85485fbaef06a2; no voters:
I20260923 15:12:19.313763 32602 leader_election.cc:288] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:12:19.313783 32607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:12:19.313921 32607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [term 1 LEADER]: Becoming Leader. State: Replica: 02b49cedffa444da9e85485fbaef06a2, State: Running, Role: LEADER
I20260923 15:12:19.314003 32607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b49cedffa444da9e85485fbaef06a2" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44249 } }
I20260923 15:12:19.314062 32602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:12:19.314306 32607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02b49cedffa444da9e85485fbaef06a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b49cedffa444da9e85485fbaef06a2" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44249 } } }
I20260923 15:12:19.314388 32607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [sys.catalog]: This master's current role is: LEADER
I20260923 15:12:19.314522 32607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b49cedffa444da9e85485fbaef06a2. Latest consensus state: current_term: 1 leader_uuid: "02b49cedffa444da9e85485fbaef06a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b49cedffa444da9e85485fbaef06a2" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 44249 } } }
I20260923 15:12:19.314576 32607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2 [sys.catalog]: This master's current role is: LEADER
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
W20260923 15:12:19.315924 32623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:12:19.316027 32623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:12:19.316114 32612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:12:19.316278 32612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:12:19.317827 32612 catalog_manager.cc:1347] Generated new cluster ID: 5f9b4919997440cd82fa170b48ae976a
I20260923 15:12:19.317907 32612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:12:19.319401 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.308987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.21.204.126:44249)
0923 15:12:19.309137 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.309144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.309169 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:12:19.309588 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.309593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.309670 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.309830 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:19.310488 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.311408 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.312145 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.312596 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.312696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.315537 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:19.315541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:19.315542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:19.315554 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:19.317594 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.318042 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.318046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.318046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.318149 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.318394 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.318395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.318396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.318695 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:19.318863 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.318944 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.319185 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260923 15:12:19.324749 32612 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:12:19.325241 32612 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:12:19.334290 32612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2: Generated new TSK 0
I20260923 15:12:19.334479 32612 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:12:19.370649 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44249
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:19.458837 32628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:19.459120 32628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:19.459163 32628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:19.460505 32628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:19.460583 32628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:19.460635 32628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:12:19.462417 32628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:19.462716 32628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:19.462987 32628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:19.468822 32636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:19.468909 32628 server_base.cc:1034] running on GCE node
W20260923 15:12:19.468852 32634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:19.469025 32633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:19.469247 32628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:19.469525 32628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:19.470708 32628 hybrid_clock.cc:648] HybridClock initialized: now 1790176339470691 us; error 26 us; skew 500 ppm
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:19.473361 32628 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:12:19.473940 32628 webserver.cc:460] Webserver started at http://127.21.204.65:33003/ using document root <none> and password file <none>
I20260923 15:12:19.474185 32628 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:19.474258 32628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:19.474421 32628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:19.475551 32628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "89d53c045b1a4519b94469e29263b3fb"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "19af11e8cab42cbeb58facbf414de2cc"
server_key_iv: "56e844a7c255fb3f01fbe7d494907dac"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.475972 32628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "89d53c045b1a4519b94469e29263b3fb"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "19af11e8cab42cbeb58facbf414de2cc"
server_key_iv: "56e844a7c255fb3f01fbe7d494907dac"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.477415 32628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:12:19.478246 32643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.478422 32628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.478536 32628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "89d53c045b1a4519b94469e29263b3fb"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "19af11e8cab42cbeb58facbf414de2cc"
server_key_iv: "56e844a7c255fb3f01fbe7d494907dac"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.478624 32628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:19.512151 32628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:19.514881 32628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:19.515056 32628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:19.515381 32628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:19.515803 32628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:19.515888 32628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.515936 32628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:19.515973 32628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.523005 32628 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:39541
I20260923 15:12:19.523061 32756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:39541 every 8 connection(s)
I20260923 15:12:19.523375 32628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:12:19.526603 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 32628
I20260923 15:12:19.526718 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:12:19.526898 22321 external_mini_cluster.cc:1428] Setting key 33853bc2e09e06949fa586956b67c8e6
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:12:19.533908 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.524120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:52175 (local address 127.21.204.126:44249)
0923 15:12:19.524195 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.524214 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.524429 (+ 215us) server_negotiation.cc:407] Connection header received
0923 15:12:19.524993 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.524994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.525007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.525069 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:19.526187 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.526851 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.528242 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.528477 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.528572 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.530864 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:19.530866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:19.530866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:19.530874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:19.532385 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.532831 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.532833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.532833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.532899 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.533189 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.533190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.533190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.533311 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:19.533393 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.533681 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.533772 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 15:12:19.534108 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.524073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:44249 (local address 127.21.204.65:52175)
0923 15:12:19.524283 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.524341 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.524882 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.525145 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.525150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.525752 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:19.526069 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.526075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.526954 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.526955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.528144 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.528147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.528586 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.528587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.528728 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.529268 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:19.529277 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:19.530747 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:19.532495 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:19.532504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:19.532506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:19.532741 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:19.532984 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:19.532985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:19.532986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:19.533104 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:19.533383 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:19.533391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:19.533538 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.533658 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.533826 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":112,"threads_started":1}
I20260923 15:12:19.534641 32757 heartbeater.cc:347] Connected to a master server at 127.21.204.126:44249
I20260923 15:12:19.534739 32757 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:19.534956 32757 heartbeater.cc:511] Master 127.21.204.126:44249 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:12:19.535507 32566 ts_manager.cc:194] Registered new tserver with Master: 89d53c045b1a4519b94469e29263b3fb (127.21.204.65:39541)
I20260923 15:12:19.536193 32566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:52175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:12:19.553397 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44249
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:19.636606 32762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:19.636819 32762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:19.636845 32762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:19.638067 32762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:19.638113 32762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:19.638150 32762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:12:19.639554 32762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:19.639745 32762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:19.639923 32762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:19.645385 302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:19.645390 300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:19.645498 32762 server_base.cc:1034] running on GCE node
W20260923 15:12:19.645402 32767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:19.645980 32762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:19.646245 32762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:19.647409 32762 hybrid_clock.cc:648] HybridClock initialized: now 1790176339647399 us; error 34 us; skew 500 ppm
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:19.650007 32762 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:12:19.650493 32762 webserver.cc:460] Webserver started at http://127.21.204.66:41197/ using document root <none> and password file <none>
I20260923 15:12:19.650727 32762 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:19.650792 32762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:19.650935 32762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:19.651957 32762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "aa2b12f7974845da9d981836f0918394"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "e564e2d705bb7ea9aae1bc26dfbab101"
server_key_iv: "4d77f8fb8ee9648554dde39ce70cb2ca"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.652318 32762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "aa2b12f7974845da9d981836f0918394"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "e564e2d705bb7ea9aae1bc26dfbab101"
server_key_iv: "4d77f8fb8ee9648554dde39ce70cb2ca"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.653636 32762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:12:19.654654 309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.654855 32762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:12:19.654982 32762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "aa2b12f7974845da9d981836f0918394"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "e564e2d705bb7ea9aae1bc26dfbab101"
server_key_iv: "4d77f8fb8ee9648554dde39ce70cb2ca"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.655068 32762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:19.670385 32762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:19.672250 32762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:19.672358 32762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:19.672533 32762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:19.672861 32762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:19.672914 32762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.672936 32762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:19.672987 32762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.678922 32762 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:41259
I20260923 15:12:19.678972 422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:41259 every 8 connection(s)
I20260923 15:12:19.679421 32762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:12:19.688354 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 32762
I20260923 15:12:19.688468 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:12:19.688661 22321 external_mini_cluster.cc:1428] Setting key cf4ec8fd2f91548380cb960cf5909b2b
I20260923 15:12:19.688974 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.680253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:44291 (local address 127.21.204.126:44249)
0923 15:12:19.680316 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.680318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.680522 (+ 204us) server_negotiation.cc:407] Connection header received
0923 15:12:19.680994 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.680995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.681004 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.681067 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:19.682056 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.682637 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.683637 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.683839 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.683935 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.686115 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:19.686116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:19.686116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:19.686123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:19.687545 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.687917 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.687919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.687919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.687976 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.688275 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.688276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.688276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.688398 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:19.688475 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.688789 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.688885 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:12:19.689152 425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.680235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:44249 (local address 127.21.204.66:44291)
0923 15:12:19.680386 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.680428 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.680913 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.681119 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.681122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.681766 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:19.681979 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.681984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.682741 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.682743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.683540 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.683542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.683942 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.683943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.684065 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.684580 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:19.684589 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:19.686008 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:19.687626 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:19.687631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:19.687633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:19.687831 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:19.688043 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:19.688043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:19.688044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:19.688171 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:19.688480 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:19.688487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:19.688648 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.688750 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.688906 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260923 15:12:19.689570 423 heartbeater.cc:347] Connected to a master server at 127.21.204.126:44249
I20260923 15:12:19.689678 423 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:19.689904 423 heartbeater.cc:511] Master 127.21.204.126:44249 requested a full tablet report, sending...
I20260923 15:12:19.690279 32566 ts_manager.cc:194] Registered new tserver with Master: aa2b12f7974845da9d981836f0918394 (127.21.204.66:41259)
I20260923 15:12:19.690646 32566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:44291
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:12:19.712996 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44249
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:12:19.792253 428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:12:19.792451 428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:12:19.792474 428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:12:19.793604 428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:12:19.793679 428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:12:19.793756 428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:12:19.795359 428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:44249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:12:19.795586 428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:12:19.795802 428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:12:19.800773 436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:19.800840 428 server_base.cc:1034] running on GCE node
W20260923 15:12:19.800815 433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:12:19.800822 434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:12:19.801198 428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:12:19.801446 428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:12:19.802588 428 hybrid_clock.cc:648] HybridClock initialized: now 1790176339802575 us; error 28 us; skew 500 ppm
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:12:19.805030 428 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:12:19.805442 428 webserver.cc:460] Webserver started at http://127.21.204.67:44639/ using document root <none> and password file <none>
I20260923 15:12:19.805667 428 fs_manager.cc:359] Metadata directory not provided
I20260923 15:12:19.805737 428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:12:19.805941 428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:12:19.807030 428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "24d20aa78f0b4798b50df2649c471101"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "0f82e78bd6df58988da249a57ddc5cea"
server_key_iv: "4b37676c3c99eb21528b5611314fb23c"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.807417 428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "24d20aa78f0b4798b50df2649c471101"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "0f82e78bd6df58988da249a57ddc5cea"
server_key_iv: "4b37676c3c99eb21528b5611314fb23c"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.808662 428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:12:19.809333 443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.809514 428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:12:19.809613 428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "24d20aa78f0b4798b50df2649c471101"
format_stamp: "Formatted at 2026-09-23 15:12:19 on dist-test-slave-t5lt"
server_key: "0f82e78bd6df58988da249a57ddc5cea"
server_key_iv: "4b37676c3c99eb21528b5611314fb23c"
server_key_version: "encryptionkey@0"
I20260923 15:12:19.809742 428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:12:19.820202 428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:12:19.822032 428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:12:19.822184 428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:12:19.822413 428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:12:19.822714 428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:12:19.822754 428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.822796 428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:12:19.822824 428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:12:19.828459 428 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:42529
I20260923 15:12:19.828531 556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:42529 every 8 connection(s)
I20260923 15:12:19.828855 428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:12:19 dist-test-slave-t5lt krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176339, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:12:19.837729 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 428
I20260923 15:12:19.837867 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:12:19.838076 22321 external_mini_cluster.cc:1428] Setting key 25a8cda1fcf572b2a788638f57f676c0
I20260923 15:12:19.838346 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.829678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:53477 (local address 127.21.204.126:44249)
0923 15:12:19.829788 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.829791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.829981 (+ 190us) server_negotiation.cc:407] Connection header received
0923 15:12:19.830544 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.830545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.830557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.830616 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:19.831570 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.832079 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.833058 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.833258 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.833331 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.835543 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:19.835544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:19.835545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:19.835552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:19.836949 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.837365 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.837365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.837366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.837425 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.837671 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.837672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.837672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.837792 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:19.837862 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.838174 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.838251 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 15:12:19.838546 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.829591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:44249 (local address 127.21.204.67:53477)
0923 15:12:19.829816 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.829861 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.830423 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.830681 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.830685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.831280 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:12:19.831470 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.831474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.832182 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.832183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.832966 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.832968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.833338 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.833339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.833471 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.834011 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:12:19.834019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:12:19.835429 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:12:19.837050 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:19.837058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:19.837060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:19.837287 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:19.837491 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:12:19.837492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:12:19.837493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:12:19.837602 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:12:19.837864 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:12:19.837871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:12:19.838041 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.838152 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.838318 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260923 15:12:19.839150 557 heartbeater.cc:347] Connected to a master server at 127.21.204.126:44249
I20260923 15:12:19.839229 557 heartbeater.cc:464] Registering TS with master...
I20260923 15:12:19.839372 557 heartbeater.cc:511] Master 127.21.204.126:44249 requested a full tablet report, sending...
I20260923 15:12:19.839728 32566 ts_manager.cc:194] Registered new tserver with Master: 24d20aa78f0b4798b50df2649c471101 (127.21.204.67:42529)
I20260923 15:12:19.840122 32566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:53477
I20260923 15:12:19.849951 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:12:19.856719 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.851213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.21.204.126:44249)
0923 15:12:19.851292 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.851294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.851351 (+ 57us) server_negotiation.cc:407] Connection header received
0923 15:12:19.851394 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.851395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.851403 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.851450 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:12:19.851974 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.852445 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.853108 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.853286 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.853333 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.854444 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:12:19.854445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:12:19.854446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:12:19.854452 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:12:19.855826 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.856103 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.856103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.856104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.856152 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:12:19.856299 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:12:19.856299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:12:19.856299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:12:19.856422 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:12:19.856492 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.856574 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.856629 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 15:12:19.858326 32566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:12:19.859871 32566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:12:19.866091 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.862216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:41259 (local address 127.0.0.1:55976)
0923 15:12:19.862530 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.862541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.862594 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.862914 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.862917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.862919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:12:19.862988 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.862990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.863975 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.863976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.864884 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.864886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.865784 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.865785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.865860 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.865895 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.865973 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.866020 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":95,"threads_started":1}
I20260923 15:12:19.866112 573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.862565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.21.204.66:41259)
0923 15:12:19.862724 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.862728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.862738 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:12:19.862776 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.862783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.862796 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.862888 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:12:19.863072 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.863872 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.864975 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.865707 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.865776 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.865832 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.865989 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.866031 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260923 15:12:19.866519 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.862193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:39541 (local address 127.0.0.1:52654)
0923 15:12:19.862355 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.862382 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.862461 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.862945 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.862946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.862949 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:12:19.862995 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.862997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.864287 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.864288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.865380 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.865383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.866301 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.866302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.866364 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.866368 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.866415 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.866455 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":44,"threads_started":1}
I20260923 15:12:19.866545 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.862343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.21.204.65:39541)
0923 15:12:19.862757 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.862761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.862770 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:12:19.862846 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.862848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.862864 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.862934 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:12:19.863078 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.864174 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.865479 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.866230 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.866297 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.866346 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.866437 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.866472 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":83,"threads_started":1}
I20260923 15:12:19.866815 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.862418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:42529 (local address 127.0.0.1:37394)
0923 15:12:19.863039 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.863049 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.863097 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.863430 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.863431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.863433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:12:19.863473 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.863475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.864664 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.864665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.865586 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.865588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.866593 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.866595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.866652 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.866657 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.866716 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.866751 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260923 15:12:19.866962 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.862528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.21.204.67:42529)
0923 15:12:19.862705 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.862710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.863156 (+ 446us) server_negotiation.cc:407] Connection header received
0923 15:12:19.863266 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.863269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.863294 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.863363 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:12:19.863541 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.864562 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.865671 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.866428 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.866502 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.866555 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.866836 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.866878 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260923 15:12:19.867110 357 tablet_service.cc:1467] Processing CreateTablet for tablet 534a6344d35b4b9eb767204de8de1da3 (DEFAULT_TABLE table=test-table [id=b9e80f2e4a864c1985e4a6c4673ae5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:12:19.867513 357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534a6344d35b4b9eb767204de8de1da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:12:19.867590 491 tablet_service.cc:1467] Processing CreateTablet for tablet 534a6344d35b4b9eb767204de8de1da3 (DEFAULT_TABLE table=test-table [id=b9e80f2e4a864c1985e4a6c4673ae5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:12:19.867861 491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534a6344d35b4b9eb767204de8de1da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:12:19.870188 574 tablet_bootstrap.cc:492] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: Bootstrap starting.
I20260923 15:12:19.870347 32691 tablet_service.cc:1467] Processing CreateTablet for tablet 534a6344d35b4b9eb767204de8de1da3 (DEFAULT_TABLE table=test-table [id=b9e80f2e4a864c1985e4a6c4673ae5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:12:19.870661 32691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534a6344d35b4b9eb767204de8de1da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:12:19.870916 574 tablet_bootstrap.cc:654] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: Neither blocks nor log segments found. Creating new log.
I20260923 15:12:19.871399 574 log.cc:824] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:19.871752 575 tablet_bootstrap.cc:492] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Bootstrap starting.
I20260923 15:12:19.872102 574 tablet_bootstrap.cc:492] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: No bootstrap required, opened a new log
I20260923 15:12:19.872154 574 ts_tablet_manager.cc:1397] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:12:19.873031 577 tablet_bootstrap.cc:492] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb: Bootstrap starting.
I20260923 15:12:19.873715 577 tablet_bootstrap.cc:654] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb: Neither blocks nor log segments found. Creating new log.
I20260923 15:12:19.873932 574 raft_consensus.cc:348] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.874127 574 raft_consensus.cc:374] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:12:19.874171 574 raft_consensus.cc:729] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2b12f7974845da9d981836f0918394, State: Initialized, Role: FOLLOWER
I20260923 15:12:19.874361 577 log.cc:824] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:19.874338 574 consensus_queue.cc:260] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.874549 575 tablet_bootstrap.cc:654] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Neither blocks nor log segments found. Creating new log.
I20260923 15:12:19.874619 574 ts_tablet_manager.cc:1428] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 15:12:19.874765 423 heartbeater.cc:503] Master 127.21.204.126:44249 was elected leader, sending a full tablet report...
I20260923 15:12:19.875067 577 tablet_bootstrap.cc:492] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb: No bootstrap required, opened a new log
I20260923 15:12:19.875077 575 log.cc:824] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Log is configured to *not* fsync() on all Append() calls
I20260923 15:12:19.875129 577 ts_tablet_manager.cc:1397] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 15:12:19.876791 577 raft_consensus.cc:348] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.876904 577 raft_consensus.cc:374] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:12:19.876924 577 raft_consensus.cc:729] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d53c045b1a4519b94469e29263b3fb, State: Initialized, Role: FOLLOWER
I20260923 15:12:19.877003 577 consensus_queue.cc:260] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.877197 577 ts_tablet_manager.cc:1428] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 15:12:19.877277 32757 heartbeater.cc:503] Master 127.21.204.126:44249 was elected leader, sending a full tablet report...
I20260923 15:12:19.877998 575 tablet_bootstrap.cc:492] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: No bootstrap required, opened a new log
I20260923 15:12:19.878072 575 ts_tablet_manager.cc:1397] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 15:12:19.879529 575 raft_consensus.cc:348] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.879648 575 raft_consensus.cc:374] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:12:19.879669 575 raft_consensus.cc:729] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d20aa78f0b4798b50df2649c471101, State: Initialized, Role: FOLLOWER
I20260923 15:12:19.879801 575 consensus_queue.cc:260] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.880136 575 ts_tablet_manager.cc:1428] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 15:12:19.880152 557 heartbeater.cc:503] Master 127.21.204.126:44249 was elected leader, sending a full tablet report...
W20260923 15:12:19.930052 424 tablet.cc:2369] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:12:19.979027 578 raft_consensus.cc:482] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:12:19.979149 578 raft_consensus.cc:504] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.979574 578 leader_election.cc:288] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89d53c045b1a4519b94469e29263b3fb (127.21.204.65:39541), 24d20aa78f0b4798b50df2649c471101 (127.21.204.67:42529)
I20260923 15:12:19.982621 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.979873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:40955 (local address 127.21.204.65:39541)
0923 15:12:19.979938 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.979940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.980062 (+ 122us) server_negotiation.cc:407] Connection header received
0923 15:12:19.980125 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.980126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.980137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.980212 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:12:19.980371 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.980831 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.981859 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.982268 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.982353 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.982364 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.982525 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.982563 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 15:12:19.982630 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.979796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:39541 (local address 127.21.204.66:40955)
0923 15:12:19.979983 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.979990 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.980063 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.980215 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.980216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.980218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:12:19.980302 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.980304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.980920 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.980921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.981770 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.981773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.982362 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.982363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.982435 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.982444 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.982509 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.982547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":128,"threads_started":1}
I20260923 15:12:19.982692 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.979869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:46023 (local address 127.21.204.67:42529)
0923 15:12:19.979970 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:19.979972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:19.979978 (+ 6us) server_negotiation.cc:407] Connection header received
0923 15:12:19.980001 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:19.980002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:19.980009 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:19.980089 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:12:19.980262 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.980742 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.981828 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.982325 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.982419 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.982427 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:19.982586 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 15:12:19.982625 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 15:12:19.982709 425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:19.979782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:42529 (local address 127.21.204.66:46023)
0923 15:12:19.979839 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:19.979851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:12:19.979895 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:12:19.980095 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:12:19.980096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:12:19.980098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:12:19.980185 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.980186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.980825 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.980826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.981719 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:12:19.981721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:12:19.982419 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:12:19.982420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:12:19.982507 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:19.982511 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:12:19.982570 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:12:19.982609 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 15:12:19.983055 511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534a6344d35b4b9eb767204de8de1da3" candidate_uuid: "aa2b12f7974845da9d981836f0918394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d20aa78f0b4798b50df2649c471101" is_pre_election: true
I20260923 15:12:19.983238 511 raft_consensus.cc:2454] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa2b12f7974845da9d981836f0918394 in term 0.
I20260923 15:12:19.983258 32711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534a6344d35b4b9eb767204de8de1da3" candidate_uuid: "aa2b12f7974845da9d981836f0918394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d53c045b1a4519b94469e29263b3fb" is_pre_election: true
I20260923 15:12:19.983417 32711 raft_consensus.cc:2454] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa2b12f7974845da9d981836f0918394 in term 0.
I20260923 15:12:19.983497 313 leader_election.cc:302] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24d20aa78f0b4798b50df2649c471101, aa2b12f7974845da9d981836f0918394; no voters:
I20260923 15:12:19.983652 578 raft_consensus.cc:2781] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:12:19.983722 578 raft_consensus.cc:482] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:12:19.983740 578 raft_consensus.cc:3037] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:12:19.984447 578 raft_consensus.cc:504] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.984588 578 leader_election.cc:288] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [CANDIDATE]: Term 1 election: Requested vote from peers 89d53c045b1a4519b94469e29263b3fb (127.21.204.65:39541), 24d20aa78f0b4798b50df2649c471101 (127.21.204.67:42529)
I20260923 15:12:19.984727 32711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534a6344d35b4b9eb767204de8de1da3" candidate_uuid: "aa2b12f7974845da9d981836f0918394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d53c045b1a4519b94469e29263b3fb"
I20260923 15:12:19.984797 32711 raft_consensus.cc:3037] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:12:19.984763 511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534a6344d35b4b9eb767204de8de1da3" candidate_uuid: "aa2b12f7974845da9d981836f0918394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24d20aa78f0b4798b50df2649c471101"
I20260923 15:12:19.984869 511 raft_consensus.cc:3037] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:12:19.985513 32711 raft_consensus.cc:2454] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa2b12f7974845da9d981836f0918394 in term 1.
I20260923 15:12:19.985595 511 raft_consensus.cc:2454] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa2b12f7974845da9d981836f0918394 in term 1.
I20260923 15:12:19.985746 312 leader_election.cc:302] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d53c045b1a4519b94469e29263b3fb, aa2b12f7974845da9d981836f0918394; no voters:
I20260923 15:12:19.985891 578 raft_consensus.cc:2781] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:12:19.986034 578 raft_consensus.cc:686] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 1 LEADER]: Becoming Leader. State: Replica: aa2b12f7974845da9d981836f0918394, State: Running, Role: LEADER
I20260923 15:12:19.986142 578 consensus_queue.cc:237] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } }
I20260923 15:12:19.986953 32566 catalog_manager.cc:5495] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa2b12f7974845da9d981836f0918394 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "aa2b12f7974845da9d981836f0918394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa2b12f7974845da9d981836f0918394" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 41259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 } health_report { overall_health: UNKNOWN } } }
I20260923 15:12:20.012323 573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:20.009585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.21.204.66:41259)
0923 15:12:20.009662 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 15:12:20.009664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:12:20.009673 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:12:20.009717 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:12:20.009718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:12:20.009738 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:12:20.009862 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:12:20.010027 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:20.010682 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:20.011462 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:12:20.011679 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:12:20.011773 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:12:20.011952 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:12:20.012070 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:12:20.012168 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 15:12:20.012188 (+ 20us) server_negotiation.cc:299] Negotiation successful
0923 15:12:20.012208 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 15:12:20.014699 511 raft_consensus.cc:1261] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 1 FOLLOWER]: Refusing update from remote peer aa2b12f7974845da9d981836f0918394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:12:20.014750 32711 raft_consensus.cc:1261] T 534a6344d35b4b9eb767204de8de1da3 P 89d53c045b1a4519b94469e29263b3fb [term 1 FOLLOWER]: Refusing update from remote peer aa2b12f7974845da9d981836f0918394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:12:20.015112 578 consensus_queue.cc:1035] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d53c045b1a4519b94469e29263b3fb" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 39541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:12:20.015372 584 consensus_queue.cc:1035] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24d20aa78f0b4798b50df2649c471101" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 42529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:12:20.017798 587 mvcc.cc:205] Tried to move back new op lower bound from 7332562288695955456 to 7332562288583938048. Current Snapshot: MvccSnapshot[applied={T|T < 7332562288695955456}]
I20260923 15:12:20.017931 588 mvcc.cc:205] Tried to move back new op lower bound from 7332562288695955456 to 7332562288583938048. Current Snapshot: MvccSnapshot[applied={T|T < 7332562288695955456}]
I20260923 15:12:20.018270 589 mvcc.cc:205] Tried to move back new op lower bound from 7332562288695955456 to 7332562288583938048. Current Snapshot: MvccSnapshot[applied={T|T < 7332562288695955456}]
I20260923 15:12:20.021130 32565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:12:20.021243 32565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:12:20.022113 32565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02b49cedffa444da9e85485fbaef06a2: Sending DeleteTablet for 3 replicas of tablet 534a6344d35b4b9eb767204de8de1da3
I20260923 15:12:20.022475 32691 tablet_service.cc:1514] Processing DeleteTablet for tablet 534a6344d35b4b9eb767204de8de1da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:12:20 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:52654
I20260923 15:12:20.022501 491 tablet_service.cc:1514] Processing DeleteTablet for tablet 534a6344d35b4b9eb767204de8de1da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:12:20 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:37394
I20260923 15:12:20.022508 357 tablet_service.cc:1514] Processing DeleteTablet for tablet 534a6344d35b4b9eb767204de8de1da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:12:20 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:55976
I20260923 15:12:20.022760 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 32628
I20260923 15:12:20.022831 599 tablet_replica.cc:330] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: stopping tablet replica
I20260923 15:12:20.022930 599 raft_consensus.cc:2229] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:12:20.023034 599 raft_consensus.cc:2258] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:12:20.023679 599 ts_tablet_manager.cc:1905] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:12:20.025956 599 ts_tablet_manager.cc:1918] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:12:20.026044 599 log.cc:1196] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/534a6344d35b4b9eb767204de8de1da3
I20260923 15:12:20.026396 599 ts_tablet_manager.cc:1939] T 534a6344d35b4b9eb767204de8de1da3 P 24d20aa78f0b4798b50df2649c471101: Deleting consensus metadata
I20260923 15:12:20.026885 32553 catalog_manager.cc:4841] TS 24d20aa78f0b4798b50df2649c471101 (127.21.204.67:42529): tablet 534a6344d35b4b9eb767204de8de1da3 (table test-table [id=b9e80f2e4a864c1985e4a6c4673ae5b7]) successfully deleted
W20260923 15:12:20.030754 32552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:39541 (error 108)
I20260923 15:12:20.030861 600 tablet_replica.cc:330] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: stopping tablet replica
I20260923 15:12:20.030987 600 raft_consensus.cc:2229] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:12:20.031124 600 raft_consensus.cc:2258] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:12:20.031478 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:20.031347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:39541 (local address 127.0.0.1:52658)
0923 15:12:20.031401 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:20.031438 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:39541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 15:12:20.031698 600 ts_tablet_manager.cc:1905] T 534a6344d35b4b9eb767204de8de1da3 P aa2b12f7974845da9d981836f0918394: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 15:12:20.031715 32552 catalog_manager.cc:4584] TS 89d53c045b1a4519b94469e29263b3fb (127.21.204.65:39541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 534a6344d35b4b9eb767204de8de1da3: Network error: Client connection negotiation failed: client connection to 127.21.204.65:39541: connect: Connection refused (error 111)
I20260923 15:12:20.031912 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 32762
I20260923 15:12:20.036835 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 428
I20260923 15:12:20.037003 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:12:20.036831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:41259 (local address 127.0.0.1:55996)
0923 15:12:20.036923 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 15:12:20.036958 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.66:41259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 15:12:20.041507 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 32534
2026-09-23T15:12:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2018 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:12:20 dist-test-slave-t5lt krb5kdc[604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:12:20 dist-test-slave-t5lt krb5kdc[604](info): set up 2 sockets
Sep 23 15:12:20 dist-test-slave-t5lt krb5kdc[604](info): commencing operation
krb5kdc: starting...
W20260923 15:12:21.088101 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:12:21 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:12:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/rangeradmin_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/rangeradmin_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/rangerlookup_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/rangerlookup_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
I20260923 15:12:21.252799 22321 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOKY8g3/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/postgres -l logfile start
2026-09-23 15:12:24.096 UTC [634] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 15:12:24.096 UTC [634] LOG: listening on IPv4 address "127.21.204.84", port 48237
2026-09-23 15:12:24.100 UTC [634] LOG: listening on Unix socket "/tmp/.s.PGSQL.48237"
2026-09-23 15:12:24.130 UTC [637] LOG: database system was shut down at 2026-09-23 15:12:22 UTC
2026-09-23 15:12:24.134 UTC [634] LOG: database system is ready to accept connections
I20260923 15:12:25.096130 22321 mini_postgres.cc:96] Postgres bound to 48237
2026-09-23 15:12:25.106 UTC [645] FATAL: database "slave" does not exist
127.21.204.84:48237 - accepting connections
I20260923 15:12:25.107402 22321 mini_ranger.cc:161] Starting Ranger...
I20260923 15:12:25.128882 22321 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 15:12:25.698202 22321 mini_ranger.cc:87] Created ranger Postgres database
I20260923 15:12:25.698302 22321 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-admin
2026-09-23 15:12:26,210 [I] DB FLAVOR :POSTGRES
2026-09-23 15:12:26,211 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 15:12:26,211 [I] Checking connection..
2026-09-23 15:12:26,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 15:12:26,498 [I] Checking connection passed.
2026-09-23 15:12:26,498 [I] --------- Verifying version history table ---------
2026-09-23 15:12:26,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 15:12:26,769 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 15:12:26,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 15:12:27,017 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 15:12:27,018 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 15:12:27,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 15:12:27.278 UTC [709] WARNING: there is no transaction in progress
2026-09-23 15:12:27,285 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 15:12:27,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 15:12:27,524 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 15:12:27,525 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 15:12:27,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 15:12:27,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 15:12:28,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t5lt.c.gcp-upstream.internal','N') ;"
2026-09-23 15:12:28,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 15:12:28,535 [I] Table x_portal_user does not exist in database ranger
2026-09-23 15:12:28,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 15:12:28,799 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 15:12:28,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 15:12:29,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 15:12:29,295 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 15:12:29,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 15:12:30.301 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.318 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.336 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.352 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.369 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.387 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.443 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.450 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.466 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.476 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.486 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.500 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.510 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.516 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.523 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:30.530 UTC [817] WARNING: there is no transaction in progress
2026-09-23 15:12:31,214 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 15:12:31,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 15:12:31,461 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 15:12:31,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 15:12:31,702 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 15:12:31,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 15:12:31,952 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 15:12:31,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 15:12:32,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 15:12:32,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t5lt.c.gcp-upstream.internal';"
2026-09-23 15:12:32,684 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 15:12:32,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 15:12:32,922 [I] DB_PATCHES have already been applied
I20260923 15:12:32.927821 22321 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOKY8g3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 15:12:32.927886 22321 mini_ranger.cc:193] Using host: 127.21.204.84
I20260923 15:12:32.931008 22321 mini_ranger.cc:227] Ranger admin URL: http://127.21.204.84:60545
Sep 23, 2026 3:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 3:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 3:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 3:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 3:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 3:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 3:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 15:12:35.156850 22321 mini_ranger.cc:160] Time spent starting Ranger: real 10.049s user 0.000s sys 0.011s
Sep 23, 2026 3:12:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 3:12:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 3:12:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 3:12:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 3:12:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 15:12:56.289417 22321 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/rangerkms_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/rangerkms_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.204.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.204.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 15:12:56 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176376, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 15:12:56.378512 22321 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 15:12:56.387936 22321 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 15:12:57.331555 22321 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 15:12:57.331629 22321 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms
I20260923 15:12:57.331641 22321 mini_ranger_kms.cc:227] Using postgres at 127.21.204.84:48237
2026-09-23 15:12:58,003 [I] DB FLAVOR :POSTGRES
2026-09-23 15:12:58,003 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 15:12:58,003 [I] Checking connection
2026-09-23 15:12:58,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 15:12:58,252 [I] connection success
2026-09-23 15:12:58,252 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 15:12:58,252 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 15:12:58,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 15:12:58,519 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 15:12:58,519 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 15:12:58,519 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 15:12:58,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:48237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 15:12:58,803 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 15:12:59.285274 22321 mini_ranger_kms.cc:310] Created kms service
I20260923 15:12:59.450019 22321 mini_ranger_kms.cc:326] Created kudu user
I20260923 15:12:59.495301 22321 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 15:12:59 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176376, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:12:59.924175 22321 mini_ranger_kms.cc:384] Added ranger policy
I20260923 15:12:59.924283 22321 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 15:12:59.924306 22321 mini_ranger_kms.cc:240] Using host: 127.21.204.84
I20260923 15:12:59.928225 22321 mini_ranger_kms.cc:276] Ranger KMS PID: 999
I20260923 15:12:59.928349 22321 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.204.84:45999
Sep 23, 2026 3:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 3:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 3:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 3:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 3:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 3:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 15:13:02.226238 22321 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.848s user 0.014s sys 0.274s
I20260923 15:13:02.226347 22321 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 15:13:02.226426 22321 mini_ranger_kms.cc:401] 127.21.204.84:45999/kms/v1/keys
Sep 23 15:13:02 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790176382, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.204.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 15:13:02 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176382, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.204.84@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
Sep 23 15:13:03 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176382, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.204.84@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
Sep 23 15:13:05 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:13:05.447172 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34505
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34505
--ranger_config_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:45999/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:05.530606 1049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:05.530853 1049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:05.530889 1049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:05.532176 1049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:13:05.532244 1049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:05.532270 1049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:05.532303 1049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:13:05.532332 1049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:13:05.533965 1049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.204.84:45999/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34505
--ranger_config_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:05.534240 1049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:05.534463 1049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:05.540346 1057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:05.540346 1055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:05.540454 1054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:05.540529 1049 server_base.cc:1034] running on GCE node
I20260923 15:13:05.540884 1049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:05.541246 1049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:05.542424 1049 hybrid_clock.cc:648] HybridClock initialized: now 1790176385542398 us; error 44 us; skew 500 ppm
Sep 23 15:13:05 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176385, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:05.545403 1049 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:13:05.545953 1049 webserver.cc:460] Webserver started at http://127.21.204.126:45631/ using document root <none> and password file <none>
I20260923 15:13:05.546196 1049 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:05.546257 1049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:05.546424 1049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:05 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176385, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:05.612406 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "6111ed4c19874405b2efc0f2dbd96c71"
format_stamp: "Formatted at 2026-09-23 15:13:05 on dist-test-slave-t5lt"
server_key: "501dbd3f7eb5b38e5885b14d7dc06ef2"
server_key_iv: "e33a8b313320c4b56c342a4678d5ffa0"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:05.612923 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "6111ed4c19874405b2efc0f2dbd96c71"
format_stamp: "Formatted at 2026-09-23 15:13:05 on dist-test-slave-t5lt"
server_key: "501dbd3f7eb5b38e5885b14d7dc06ef2"
server_key_iv: "e33a8b313320c4b56c342a4678d5ffa0"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:05.614684 1049 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 15:13:05.629994 1064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:05.630218 1049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:05.630313 1049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "6111ed4c19874405b2efc0f2dbd96c71"
format_stamp: "Formatted at 2026-09-23 15:13:05 on dist-test-slave-t5lt"
server_key: "501dbd3f7eb5b38e5885b14d7dc06ef2"
server_key_iv: "e33a8b313320c4b56c342a4678d5ffa0"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:05.630410 1049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:05.642668 1049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:05.644737 1049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:05.644909 1049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:05.649930 1049 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34505
I20260923 15:13:05.649946 1116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34505 every 8 connection(s)
I20260923 15:13:05.650394 1049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:13:05.651186 1117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:05.653481 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 1049
I20260923 15:13:05.653638 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:13:05.653957 22321 external_mini_cluster.cc:1428] Setting key 7a379715549f99a472af9b6757ea44d8
I20260923 15:13:05.654358 1117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71: Bootstrap starting.
I20260923 15:13:05.655376 1117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:05.655723 1117 log.cc:824] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:05.656579 1117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71: No bootstrap required, opened a new log
I20260923 15:13:05.658694 1117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6111ed4c19874405b2efc0f2dbd96c71" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34505 } }
I20260923 15:13:05.658825 1117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:05.658849 1117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6111ed4c19874405b2efc0f2dbd96c71, State: Initialized, Role: FOLLOWER
I20260923 15:13:05.658986 1117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6111ed4c19874405b2efc0f2dbd96c71" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34505 } }
I20260923 15:13:05.659061 1117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:13:05.659104 1117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:13:05.659149 1117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:05.659672 1117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6111ed4c19874405b2efc0f2dbd96c71" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34505 } }
I20260923 15:13:05.659755 1117 leader_election.cc:302] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6111ed4c19874405b2efc0f2dbd96c71; no voters:
I20260923 15:13:05.659931 1117 leader_election.cc:288] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:13:05.660060 1122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:13:05.660193 1117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:13:05.660331 1122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [term 1 LEADER]: Becoming Leader. State: Replica: 6111ed4c19874405b2efc0f2dbd96c71, State: Running, Role: LEADER
I20260923 15:13:05.660519 1122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6111ed4c19874405b2efc0f2dbd96c71" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34505 } }
I20260923 15:13:05.661019 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6111ed4c19874405b2efc0f2dbd96c71. Latest consensus state: current_term: 1 leader_uuid: "6111ed4c19874405b2efc0f2dbd96c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6111ed4c19874405b2efc0f2dbd96c71" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34505 } } }
I20260923 15:13:05.661183 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:05.661785 1123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6111ed4c19874405b2efc0f2dbd96c71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6111ed4c19874405b2efc0f2dbd96c71" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34505 } } }
I20260923 15:13:05.661865 1123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71 [sys.catalog]: This master's current role is: LEADER
Sep 23 15:13:05 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:05.662220 1131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:13:05.662238 1117 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 15:13:05.665208 1131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:13:05.668267 1131 catalog_manager.cc:1347] Generated new cluster ID: 4c901dcbf7f54deca4216a8204b4e8f6
I20260923 15:13:05.668355 1131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:13:05.669451 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:05.654596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.21.204.126:34505)
0923 15:13:05.654809 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:05.654821 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:05.654860 (+ 39us) server_negotiation.cc:407] Connection header received
0923 15:13:05.655355 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:05.655363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:05.655415 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:05.655602 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:05.656537 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:05.657596 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:05.658566 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:05.659218 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:05.659359 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:05.664273 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:05.664331 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:05.664332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:05.664456 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:05.667404 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:05.667875 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:05.667878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:05.667879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:05.667977 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:05.668191 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:05.668192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:05.668192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:05.668510 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:05.668639 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:05.669110 (+ 471us) server_negotiation.cc:299] Negotiation successful
0923 15:13:05.669242 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
I20260923 15:13:05.702596 1131 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:13:05.703559 1131 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:13:05.708451 1131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71: Generated new TSK 0
I20260923 15:13:05.708643 1131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:13:06.516111 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:45999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34505
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:06.621681 1162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:06.621938 1162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:06.621977 1162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:06.623227 1162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:06.623268 1162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:06.623293 1162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:13:06.624821 1162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.204.84:45999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:06.625072 1162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:06.625317 1162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:06.632304 1168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:06.632397 1170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:06.632519 1167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:06.633013 1162 server_base.cc:1034] running on GCE node
I20260923 15:13:06.633229 1162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:06.633509 1162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:06.634691 1162 hybrid_clock.cc:648] HybridClock initialized: now 1790176386634673 us; error 31 us; skew 500 ppm
Sep 23 15:13:06 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:06.637512 1162 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:13:06.638064 1162 webserver.cc:460] Webserver started at http://127.21.204.65:38025/ using document root <none> and password file <none>
I20260923 15:13:06.638322 1162 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:06.638370 1162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:06.638492 1162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:06 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:06.651852 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03"
format_stamp: "Formatted at 2026-09-23 15:13:06 on dist-test-slave-t5lt"
server_key: "c224f632003aacb940c570febd068526"
server_key_iv: "6408cc090521fd4d3aa688a5ceb59454"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:06.652267 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03"
format_stamp: "Formatted at 2026-09-23 15:13:06 on dist-test-slave-t5lt"
server_key: "c224f632003aacb940c570febd068526"
server_key_iv: "6408cc090521fd4d3aa688a5ceb59454"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:06.655212 1162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 15:13:06.669477 1177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:06.669703 1162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:06.669796 1162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03"
format_stamp: "Formatted at 2026-09-23 15:13:06 on dist-test-slave-t5lt"
server_key: "c224f632003aacb940c570febd068526"
server_key_iv: "6408cc090521fd4d3aa688a5ceb59454"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:06.669914 1162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:06.694350 1162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:06.697623 1162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:06.697831 1162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:06.698230 1162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:06.698709 1162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:06.698760 1162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:06.698791 1162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:06.698808 1162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:06.708961 1162 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:37061
I20260923 15:13:06.709482 1162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:13:06.710922 1290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:37061 every 8 connection(s)
I20260923 15:13:06.711828 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 1162
I20260923 15:13:06.711941 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:13:06.712141 22321 external_mini_cluster.cc:1428] Setting key e80edc182a1086936aef5ad4972caf0c
Sep 23 15:13:06 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
I20260923 15:13:06.729036 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:06.711658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34505 (local address 127.21.204.65:40921)
0923 15:13:06.717652 (+ 5994us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:06.717706 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:06.718709 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:06.719025 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:06.719030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:06.719548 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:06.719984 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:06.719992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.721056 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.721057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:06.722323 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:06.722327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.722847 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.722848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:06.722989 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:06.723523 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:06.723531 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:06.725083 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:06.727143 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:06.727152 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:06.727154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:06.727437 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:06.727686 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:06.727687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:06.727688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:06.727847 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:06.728215 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:06.728222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:06.728393 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 15:13:06.728495 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 15:13:06.728656 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5912,"thread_start_us":99,"threads_started":1}
I20260923 15:13:06.729957 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:06.711925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:40921 (local address 127.21.204.126:34505)
0923 15:13:06.717942 (+ 6017us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:06.717945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:06.717956 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:13:06.718827 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:06.718828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:06.718845 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:06.718935 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:06.720109 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.720925 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.722422 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.722734 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.722834 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:06.725187 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:06.725189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:06.725190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:06.725197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:06.727028 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:06.727536 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:06.727537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:06.727537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:06.727605 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:06.727935 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:06.727936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:06.727937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:06.728134 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:06.729630 (+ 1496us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:06.729680 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 15:13:06.729779 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5992,"thread_start_us":69,"threads_started":1}
I20260923 15:13:06.730715 1291 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34505
I20260923 15:13:06.730829 1291 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:06.731024 1291 heartbeater.cc:511] Master 127.21.204.126:34505 requested a full tablet report, sending...
I20260923 15:13:06.731725 1081 ts_manager.cc:194] Registered new tserver with Master: 3ccdd4ecc69b4eb484d7784ad1bc2e03 (127.21.204.65:37061)
I20260923 15:13:06.732797 1081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:40921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:13:06.748998 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:45999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34505
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:06.875164 1299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:06.875452 1299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:06.875492 1299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:06.877231 1299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:06.877291 1299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:06.877332 1299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:13:06.879472 1299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.204.84:45999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:06.879710 1299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:06.880033 1299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:06.886407 1309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:06.886494 1307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:06.886842 1306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:06.887243 1299 server_base.cc:1034] running on GCE node
I20260923 15:13:06.887535 1299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:06.887858 1299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:06.889066 1299 hybrid_clock.cc:648] HybridClock initialized: now 1790176386889056 us; error 63 us; skew 500 ppm
Sep 23 15:13:06 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:06.892204 1299 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:13:06.892720 1299 webserver.cc:460] Webserver started at http://127.21.204.66:42213/ using document root <none> and password file <none>
I20260923 15:13:06.893035 1299 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:06.893108 1299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:06.893296 1299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:06 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:06.910001 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "468be9f750bc4e9eb9057b40f7952b37"
format_stamp: "Formatted at 2026-09-23 15:13:06 on dist-test-slave-t5lt"
server_key: "a01aa6b68906fbaa84b21b95c2be9550"
server_key_iv: "5490818444590f168bff1cdab474847c"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:06.910562 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "468be9f750bc4e9eb9057b40f7952b37"
format_stamp: "Formatted at 2026-09-23 15:13:06 on dist-test-slave-t5lt"
server_key: "a01aa6b68906fbaa84b21b95c2be9550"
server_key_iv: "5490818444590f168bff1cdab474847c"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:06.913534 1299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260923 15:13:06.929131 1316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:06.929374 1299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 15:13:06.929477 1299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "468be9f750bc4e9eb9057b40f7952b37"
format_stamp: "Formatted at 2026-09-23 15:13:06 on dist-test-slave-t5lt"
server_key: "a01aa6b68906fbaa84b21b95c2be9550"
server_key_iv: "5490818444590f168bff1cdab474847c"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:06.929586 1299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 23 15:13:06 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790176386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:06.949287 1299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:06.951807 1299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:06.952020 1299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:06.952311 1299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:06.952741 1299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:06.952831 1299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:06.952908 1299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:06.952970 1299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:06.960538 1299 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:37281
I20260923 15:13:06.961148 1299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:13:06.962831 1429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:37281 every 8 connection(s)
I20260923 15:13:06.967882 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 1299
I20260923 15:13:06.967995 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:13:06.968183 22321 external_mini_cluster.cc:1428] Setting key 8a308c9ca32cd180ae9831bfe894bf7a
Sep 23 15:13:06 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176386, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:06.983350 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:06.964366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:59321 (local address 127.21.204.126:34505)
0923 15:13:06.964414 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:06.964416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:06.964662 (+ 246us) server_negotiation.cc:407] Connection header received
0923 15:13:06.965596 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:06.965597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:06.965614 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:06.965645 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:06.966842 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.967650 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.975157 (+ 7507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.975479 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.975564 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:06.979602 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:06.979604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:06.979605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:06.979613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:06.981526 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:06.982112 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:06.982113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:06.982114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:06.982178 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:06.982510 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:06.982511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:06.982511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:06.982650 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:06.982704 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:06.983161 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 15:13:06.983251 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:13:06.984104 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:06.964092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34505 (local address 127.21.204.66:59321)
0923 15:13:06.964525 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:06.964573 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:06.965382 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:06.965827 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:06.965832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:06.966300 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:06.966726 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:06.966733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.968652 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.968653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:06.975009 (+ 6356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:06.975033 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:06.976859 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:06.976865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:06.977011 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:06.977668 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:06.977681 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:06.979467 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:06.981636 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:06.981642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:06.981645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:06.981993 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:06.982276 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:06.982277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:06.982277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:06.982418 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:06.982741 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:06.982748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:06.982962 (+ 214us) client_negotiation.cc:769] Sending connection context
0923 15:13:06.983694 (+ 732us) client_negotiation.cc:240] Negotiation successful
0923 15:13:06.983869 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":78,"threads_started":1}
I20260923 15:13:06.984707 1430 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34505
I20260923 15:13:06.984802 1430 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:06.984977 1430 heartbeater.cc:511] Master 127.21.204.126:34505 requested a full tablet report, sending...
I20260923 15:13:06.985343 1081 ts_manager.cc:194] Registered new tserver with Master: 468be9f750bc4e9eb9057b40f7952b37 (127.21.204.66:37281)
I20260923 15:13:06.985839 1081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:59321
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:13:07.006628 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:45999/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34505
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:46777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:07.102763 1435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:07.103176 1435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:07.103250 1435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:07.104534 1435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:07.104604 1435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:07.104673 1435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:13:07.106839 1435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:46777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.204.84:45999/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:07.107218 1435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:07.107503 1435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:07.112964 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:07.113217 1444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:07.113363 1441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:07.113602 1435 server_base.cc:1034] running on GCE node
I20260923 15:13:07.113791 1435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:07.114053 1435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:07.121456 1435 hybrid_clock.cc:648] HybridClock initialized: now 1790176387121425 us; error 37 us; skew 500 ppm
Sep 23 15:13:07 dist-test-slave-t5lt krb5kdc[604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:07.124931 1435 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:13:07.125430 1435 webserver.cc:460] Webserver started at http://127.21.204.67:34779/ using document root <none> and password file <none>
I20260923 15:13:07.125680 1435 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:07.125728 1435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:07.125847 1435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:07 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:07.142418 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "131a1952b4fb46dc97f7737aa0b478ac"
format_stamp: "Formatted at 2026-09-23 15:13:07 on dist-test-slave-t5lt"
server_key: "28b2941e4490d9d8f50ce166ffb94339"
server_key_iv: "f0d5e6e7f1393788d6300d4ea5a97d46"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:07.143054 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "131a1952b4fb46dc97f7737aa0b478ac"
format_stamp: "Formatted at 2026-09-23 15:13:07 on dist-test-slave-t5lt"
server_key: "28b2941e4490d9d8f50ce166ffb94339"
server_key_iv: "f0d5e6e7f1393788d6300d4ea5a97d46"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:07.154187 1435 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260923 15:13:07.167846 1451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:07.168052 1435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:07.168156 1435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "131a1952b4fb46dc97f7737aa0b478ac"
format_stamp: "Formatted at 2026-09-23 15:13:07 on dist-test-slave-t5lt"
server_key: "28b2941e4490d9d8f50ce166ffb94339"
server_key_iv: "f0d5e6e7f1393788d6300d4ea5a97d46"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:07.168263 1435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:07.204613 1435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:07.207512 1435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:07.207731 1435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:07.208002 1435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:07.208460 1435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:07.208556 1435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:07.208632 1435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:07.208705 1435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:07.221084 1435 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:38157
I20260923 15:13:07.221621 1435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:13:07.226191 1564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:38157 every 8 connection(s)
I20260923 15:13:07.228227 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 1435
I20260923 15:13:07.228384 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:13:07.228646 22321 external_mini_cluster.cc:1428] Setting key 0298be346ebaf3f2df26cb4cd5936913
Sep 23 15:13:07 dist-test-slave-t5lt krb5kdc[604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176387, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:07.235814 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.222578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:39209 (local address 127.21.204.126:34505)
0923 15:13:07.222633 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.222635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.223076 (+ 441us) server_negotiation.cc:407] Connection header received
0923 15:13:07.224244 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.224246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.224270 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.224384 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:07.225648 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.226723 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.228239 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.228536 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.228585 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.231636 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:07.231638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:07.231638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:07.231647 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:07.233607 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:07.234314 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:07.234316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:07.234316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:07.234380 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:07.234770 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:07.234772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:07.234772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:07.234967 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:07.235026 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.235501 (+ 475us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.235592 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:13:07.235935 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.222525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34505 (local address 127.21.204.67:39209)
0923 15:13:07.222883 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.222933 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.224100 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.224472 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.224477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.225005 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:07.225528 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.225535 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.226856 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.226857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.228129 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.228132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.228687 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.228688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.228839 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.229460 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:07.229471 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:07.231508 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:07.233727 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:07.233744 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:07.233746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:07.234175 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:07.234460 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:07.234461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:07.234462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:07.234679 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:07.235100 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:07.235107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:07.235313 (+ 206us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.235453 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.235642 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":176,"threads_started":1}
I20260923 15:13:07.236707 1565 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34505
I20260923 15:13:07.236814 1565 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:07.237057 1565 heartbeater.cc:511] Master 127.21.204.126:34505 requested a full tablet report, sending...
I20260923 15:13:07.237577 1081 ts_manager.cc:194] Registered new tserver with Master: 131a1952b4fb46dc97f7737aa0b478ac (127.21.204.67:38157)
I20260923 15:13:07.238293 1081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:39209
I20260923 15:13:07.241183 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:13:07.252111 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.243504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.21.204.126:34505)
0923 15:13:07.243544 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.243546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.243555 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:13:07.243653 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.243654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.243663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.243726 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:07.244641 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.245368 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.246660 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.246949 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.247028 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.248850 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:07.248851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:07.248852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:07.248859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:07.250764 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:07.251243 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:07.251244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:07.251245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:07.251306 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:07.251544 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:07.251545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:07.251546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:07.251673 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:07.251775 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.251949 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.252024 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:13:07.254350 1081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:13:07.256875 1081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:13:07.266356 1580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.260466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:38157 (local address 127.0.0.1:40708)
0923 15:13:07.261268 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.261287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.261386 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.261713 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.261716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.261719 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:07.261784 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.261786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.263398 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.263399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.264663 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.264666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.266067 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.266068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.266149 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.266162 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.266231 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.266280 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":121,"threads_started":1}
I20260923 15:13:07.266922 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.261008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.21.204.67:38157)
0923 15:13:07.261272 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.261277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.261350 (+ 73us) server_negotiation.cc:407] Connection header received
0923 15:13:07.261514 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.261516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.261534 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.261616 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:07.261868 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.263311 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.264787 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.265861 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.265952 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.266031 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.266797 (+ 766us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.266840 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260923 15:13:07.267489 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.260689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:37281 (local address 127.0.0.1:36378)
0923 15:13:07.261594 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.261612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.261670 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.262464 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.262465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.262468 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:07.262518 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.262519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.264580 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.264581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.265774 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.265777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.267219 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.267220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.267295 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.267300 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.267334 (+ 34us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.267384 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":73,"threads_started":1}
I20260923 15:13:07.267741 1499 tablet_service.cc:1467] Processing CreateTablet for tablet 2810224c964442f9ada5960ac67ec2f5 (DEFAULT_TABLE table=test-table [id=8947ee31eea74662ad503bcadf20d79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:13:07.268142 1499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2810224c964442f9ada5960ac67ec2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:07.268339 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.260901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.21.204.66:37281)
0923 15:13:07.262146 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.262160 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.262171 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:13:07.262262 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.262263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.262297 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.262600 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:07.262760 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.264370 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.266066 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.267101 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.267856 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.268087 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.268116 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.268153 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134,"thread_start_us":140,"threads_started":1}
I20260923 15:13:07.269173 1363 tablet_service.cc:1467] Processing CreateTablet for tablet 2810224c964442f9ada5960ac67ec2f5 (DEFAULT_TABLE table=test-table [id=8947ee31eea74662ad503bcadf20d79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:13:07.269500 1363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2810224c964442f9ada5960ac67ec2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:07.272882 1584 tablet_bootstrap.cc:492] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Bootstrap starting.
I20260923 15:13:07.274082 1584 tablet_bootstrap.cc:654] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:07.274340 1583 tablet_bootstrap.cc:492] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Bootstrap starting.
I20260923 15:13:07.274498 1584 log.cc:824] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:07.276183 1584 tablet_bootstrap.cc:492] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: No bootstrap required, opened a new log
I20260923 15:13:07.276259 1584 ts_tablet_manager.cc:1397] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:13:07.276288 1583 tablet_bootstrap.cc:654] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:07.276612 1583 log.cc:824] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:07.278224 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.260341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37061 (local address 127.0.0.1:59814)
0923 15:13:07.260639 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.260801 (+ 162us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.260861 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.274168 (+ 13307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.274169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.274172 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:07.274220 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.274221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.275678 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.275680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.276969 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.276973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.277994 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.277995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.278065 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.278070 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.278103 (+ 33us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.278146 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":80,"threads_started":1}
I20260923 15:13:07.278613 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.260614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.21.204.65:37061)
0923 15:13:07.273978 (+ 13364us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.273983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.273995 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:13:07.274032 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.274034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.274053 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.274156 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:07.274331 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.275576 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.277101 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.277846 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.278387 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.278462 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.278488 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.278545 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13311,"thread_start_us":52,"threads_started":1}
I20260923 15:13:07.278957 1584 raft_consensus.cc:348] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.279174 1584 raft_consensus.cc:374] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:07.279210 1584 raft_consensus.cc:729] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468be9f750bc4e9eb9057b40f7952b37, State: Initialized, Role: FOLLOWER
I20260923 15:13:07.279294 1225 tablet_service.cc:1467] Processing CreateTablet for tablet 2810224c964442f9ada5960ac67ec2f5 (DEFAULT_TABLE table=test-table [id=8947ee31eea74662ad503bcadf20d79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:13:07.279633 1225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2810224c964442f9ada5960ac67ec2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:07.280092 1584 consensus_queue.cc:260] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.280524 1584 ts_tablet_manager.cc:1428] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260923 15:13:07.280694 1430 heartbeater.cc:503] Master 127.21.204.126:34505 was elected leader, sending a full tablet report...
I20260923 15:13:07.282869 1588 tablet_bootstrap.cc:492] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03: Bootstrap starting.
I20260923 15:13:07.283648 1588 tablet_bootstrap.cc:654] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:07.284011 1588 log.cc:824] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:07.284802 1588 tablet_bootstrap.cc:492] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03: No bootstrap required, opened a new log
I20260923 15:13:07.284884 1588 ts_tablet_manager.cc:1397] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:13:07.287426 1588 raft_consensus.cc:348] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.287582 1588 raft_consensus.cc:374] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:07.287612 1588 raft_consensus.cc:729] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdd4ecc69b4eb484d7784ad1bc2e03, State: Initialized, Role: FOLLOWER
I20260923 15:13:07.288313 1588 consensus_queue.cc:260] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.288689 1588 ts_tablet_manager.cc:1428] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 15:13:07.288851 1291 heartbeater.cc:503] Master 127.21.204.126:34505 was elected leader, sending a full tablet report...
I20260923 15:13:07.294047 1583 tablet_bootstrap.cc:492] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: No bootstrap required, opened a new log
I20260923 15:13:07.294135 1583 ts_tablet_manager.cc:1397] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20260923 15:13:07.296300 1583 raft_consensus.cc:348] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.296474 1583 raft_consensus.cc:374] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:07.296509 1583 raft_consensus.cc:729] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131a1952b4fb46dc97f7737aa0b478ac, State: Initialized, Role: FOLLOWER
I20260923 15:13:07.296619 1583 consensus_queue.cc:260] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.296933 1583 ts_tablet_manager.cc:1428] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:13:07.297199 1565 heartbeater.cc:503] Master 127.21.204.126:34505 was elected leader, sending a full tablet report...
I20260923 15:13:07.328737 1587 raft_consensus.cc:482] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:13:07.329032 1587 raft_consensus.cc:504] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.329339 1591 raft_consensus.cc:482] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:13:07.329442 1591 raft_consensus.cc:504] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.329514 1587 leader_election.cc:288] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccdd4ecc69b4eb484d7784ad1bc2e03 (127.21.204.65:37061), 131a1952b4fb46dc97f7737aa0b478ac (127.21.204.67:38157)
I20260923 15:13:07.329785 1591 leader_election.cc:288] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccdd4ecc69b4eb484d7784ad1bc2e03 (127.21.204.65:37061), 468be9f750bc4e9eb9057b40f7952b37 (127.21.204.66:37281)
I20260923 15:13:07.334347 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.330011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:37281 (local address 127.21.204.67:34703)
0923 15:13:07.330064 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.330076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.330115 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.330905 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.330907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.330909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:07.331073 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.331075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.332083 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.332085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.333264 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.333267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.334102 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.334103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.334178 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.334182 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.334233 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.334272 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 15:13:07.334717 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.329852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:38157 (local address 127.21.204.66:52589)
0923 15:13:07.330198 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.330209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.330258 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.330999 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.331000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.331002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:07.331148 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.331150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.332375 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.332376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.333550 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.333553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.334371 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.334372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.334522 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.334526 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.334586 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.334625 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20260923 15:13:07.334843 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.330610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:34703 (local address 127.21.204.66:37281)
0923 15:13:07.330651 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.330653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.330662 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:13:07.330686 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.330687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.330696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.330753 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:07.331171 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.331965 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.333373 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.333951 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.334020 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.334030 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.334735 (+ 705us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.334775 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:13:07.335678 1367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2810224c964442f9ada5960ac67ec2f5" candidate_uuid: "131a1952b4fb46dc97f7737aa0b478ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468be9f750bc4e9eb9057b40f7952b37" is_pre_election: true
I20260923 15:13:07.335803 1367 raft_consensus.cc:2454] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 131a1952b4fb46dc97f7737aa0b478ac in term 0.
I20260923 15:13:07.336249 1454 leader_election.cc:302] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 131a1952b4fb46dc97f7737aa0b478ac, 468be9f750bc4e9eb9057b40f7952b37; no voters:
I20260923 15:13:07.336639 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.330815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:52589 (local address 127.21.204.67:38157)
0923 15:13:07.330858 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.330860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.330867 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:13:07.330888 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.330888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.330896 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.330942 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:07.331230 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.332040 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.333676 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.334245 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.334315 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.334323 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.336529 (+ 2206us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.336580 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":25}
I20260923 15:13:07.336925 1591 raft_consensus.cc:2781] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:13:07.336994 1591 raft_consensus.cc:482] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:13:07.337018 1591 raft_consensus.cc:3037] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:07.337172 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2810224c964442f9ada5960ac67ec2f5" candidate_uuid: "468be9f750bc4e9eb9057b40f7952b37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131a1952b4fb46dc97f7737aa0b478ac" is_pre_election: true
I20260923 15:13:07.337970 1591 raft_consensus.cc:504] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.338119 1591 leader_election.cc:288] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [CANDIDATE]: Term 1 election: Requested vote from peers 3ccdd4ecc69b4eb484d7784ad1bc2e03 (127.21.204.65:37061), 468be9f750bc4e9eb9057b40f7952b37 (127.21.204.66:37281)
I20260923 15:13:07.338121 1519 raft_consensus.cc:2379] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 468be9f750bc4e9eb9057b40f7952b37 in current term 1: Already voted for candidate 131a1952b4fb46dc97f7737aa0b478ac in this term.
I20260923 15:13:07.338505 1367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2810224c964442f9ada5960ac67ec2f5" candidate_uuid: "131a1952b4fb46dc97f7737aa0b478ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468be9f750bc4e9eb9057b40f7952b37"
I20260923 15:13:07.338575 1367 raft_consensus.cc:3037] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:07.339521 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.330282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37061 (local address 127.21.204.67:43279)
0923 15:13:07.334419 (+ 4137us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.334430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.334469 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.334916 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.334918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.334919 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:07.334949 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.334950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.335838 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.335839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.336825 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.336827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.338949 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.338950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.339380 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.339383 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.339430 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.339462 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4127,"thread_start_us":167,"threads_started":1}
I20260923 15:13:07.339591 1367 raft_consensus.cc:2454] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131a1952b4fb46dc97f7737aa0b478ac in term 1.
I20260923 15:13:07.339773 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.330645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:43279 (local address 127.21.204.65:37061)
0923 15:13:07.330863 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.330865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.334521 (+ 3656us) server_negotiation.cc:407] Connection header received
0923 15:13:07.334550 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.334551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.334560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.334601 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:07.335026 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.335716 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.338252 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.338838 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.338910 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.338918 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.339662 (+ 744us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.339711 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260923 15:13:07.339794 1454 leader_election.cc:302] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 131a1952b4fb46dc97f7737aa0b478ac, 468be9f750bc4e9eb9057b40f7952b37; no voters:
I20260923 15:13:07.340158 1245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2810224c964442f9ada5960ac67ec2f5" candidate_uuid: "131a1952b4fb46dc97f7737aa0b478ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" is_pre_election: true
I20260923 15:13:07.340340 1245 raft_consensus.cc:2454] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 131a1952b4fb46dc97f7737aa0b478ac in term 0.
I20260923 15:13:07.340466 1244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2810224c964442f9ada5960ac67ec2f5" candidate_uuid: "131a1952b4fb46dc97f7737aa0b478ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03"
I20260923 15:13:07.340538 1244 raft_consensus.cc:3037] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:07.341362 1244 raft_consensus.cc:2454] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131a1952b4fb46dc97f7737aa0b478ac in term 1.
I20260923 15:13:07.341814 1591 raft_consensus.cc:2781] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:13:07.341996 1591 raft_consensus.cc:686] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 1 LEADER]: Becoming Leader. State: Replica: 131a1952b4fb46dc97f7737aa0b478ac, State: Running, Role: LEADER
I20260923 15:13:07.342094 1591 consensus_queue.cc:237] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } }
I20260923 15:13:07.343000 1081 catalog_manager.cc:5495] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 131a1952b4fb46dc97f7737aa0b478ac (127.21.204.67). New cstate: current_term: 1 leader_uuid: "131a1952b4fb46dc97f7737aa0b478ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "131a1952b4fb46dc97f7737aa0b478ac" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 } health_report { overall_health: UNKNOWN } } }
I20260923 15:13:07.346225 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.330419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37061 (local address 127.21.204.66:42531)
0923 15:13:07.334788 (+ 4369us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.334798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:07.334837 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:07.341684 (+ 6847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:07.341685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:07.341687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:07.341725 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.341726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.344159 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.344160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.345219 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:07.345222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.346008 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.346009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:07.346069 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.346073 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:13:07.346118 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:13:07.346153 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4360,"thread_start_us":165,"threads_started":1}
I20260923 15:13:07.346628 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.330685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:42531 (local address 127.21.204.65:37061)
0923 15:13:07.339864 (+ 9179us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.339866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.339873 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:13:07.339891 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.339892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.339900 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.339953 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:07.343406 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.344062 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.345320 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.345846 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.345976 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.345986 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.346527 (+ 541us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.346568 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9159,"thread_start_us":141,"threads_started":1}
I20260923 15:13:07.346913 1244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2810224c964442f9ada5960ac67ec2f5" candidate_uuid: "468be9f750bc4e9eb9057b40f7952b37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" is_pre_election: true
I20260923 15:13:07.347002 1244 raft_consensus.cc:2379] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 468be9f750bc4e9eb9057b40f7952b37 in current term 1: Already voted for candidate 131a1952b4fb46dc97f7737aa0b478ac in this term.
I20260923 15:13:07.347266 1320 leader_election.cc:302] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 468be9f750bc4e9eb9057b40f7952b37; no voters: 131a1952b4fb46dc97f7737aa0b478ac, 3ccdd4ecc69b4eb484d7784ad1bc2e03
I20260923 15:13:07.347448 1587 raft_consensus.cc:2726] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 15:13:07.364357 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.361012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.21.204.67:38157)
0923 15:13:07.361054 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:07.361057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:07.361067 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:13:07.361140 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:07.361141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:07.361163 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:07.361237 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:13:07.361419 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.362186 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.363273 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:07.363538 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:07.363596 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:07.363780 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:13:07.363929 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:13:07.364011 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:07.364240 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 15:13:07.364258 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:13:07.366947 1367 raft_consensus.cc:1261] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 1 FOLLOWER]: Refusing update from remote peer 131a1952b4fb46dc97f7737aa0b478ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:13:07.367324 1595 consensus_queue.cc:1035] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "468be9f750bc4e9eb9057b40f7952b37" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 37281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:13:07.371007 1244 raft_consensus.cc:1261] T 2810224c964442f9ada5960ac67ec2f5 P 3ccdd4ecc69b4eb484d7784ad1bc2e03 [term 1 FOLLOWER]: Refusing update from remote peer 131a1952b4fb46dc97f7737aa0b478ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:13:07.371359 1595 consensus_queue.cc:1035] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdd4ecc69b4eb484d7784ad1bc2e03" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:13:07.377938 1598 mvcc.cc:205] Tried to move back new op lower bound from 7332562482651848704 to 7332562482553815040. Current Snapshot: MvccSnapshot[applied={T|T < 7332562482651848704}]
I20260923 15:13:07.378296 1601 mvcc.cc:205] Tried to move back new op lower bound from 7332562482651848704 to 7332562482553815040. Current Snapshot: MvccSnapshot[applied={T|T < 7332562482651848704}]
I20260923 15:13:07.382336 1599 mvcc.cc:205] Tried to move back new op lower bound from 7332562482651848704 to 7332562482553815040. Current Snapshot: MvccSnapshot[applied={T|T < 7332562482651848704}]
I20260923 15:13:07.384776 1081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:13:07.384876 1081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:13:07.385636 1081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6111ed4c19874405b2efc0f2dbd96c71: Sending DeleteTablet for 3 replicas of tablet 2810224c964442f9ada5960ac67ec2f5
I20260923 15:13:07.385996 1225 tablet_service.cc:1514] Processing DeleteTablet for tablet 2810224c964442f9ada5960ac67ec2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:13:07 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:59814
I20260923 15:13:07.386427 1499 tablet_service.cc:1514] Processing DeleteTablet for tablet 2810224c964442f9ada5960ac67ec2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:13:07 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:40708
I20260923 15:13:07.386626 1363 tablet_service.cc:1514] Processing DeleteTablet for tablet 2810224c964442f9ada5960ac67ec2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:13:07 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:36378
I20260923 15:13:07.387203 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 1162
I20260923 15:13:07.398052 1608 tablet_replica.cc:330] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: stopping tablet replica
I20260923 15:13:07.398238 1609 tablet_replica.cc:330] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: stopping tablet replica
I20260923 15:13:07.398460 1609 raft_consensus.cc:2229] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:13:07.398607 1608 raft_consensus.cc:2229] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:13:07.398737 1609 raft_consensus.cc:2258] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:13:07.398924 1608 raft_consensus.cc:2258] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:13:07.399538 1608 ts_tablet_manager.cc:1905] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:13:07.406373 1608 ts_tablet_manager.cc:1918] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:13:07.406772 1608 log.cc:1196] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/2810224c964442f9ada5960ac67ec2f5
I20260923 15:13:07.407214 1609 ts_tablet_manager.cc:1905] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:13:07.409081 1609 ts_tablet_manager.cc:1918] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:13:07.409446 1608 ts_tablet_manager.cc:1939] T 2810224c964442f9ada5960ac67ec2f5 P 131a1952b4fb46dc97f7737aa0b478ac: Deleting consensus metadata
I20260923 15:13:07.409567 1609 log.cc:1196] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/2810224c964442f9ada5960ac67ec2f5
I20260923 15:13:07.409910 1609 ts_tablet_manager.cc:1939] T 2810224c964442f9ada5960ac67ec2f5 P 468be9f750bc4e9eb9057b40f7952b37: Deleting consensus metadata
I20260923 15:13:07.410398 1067 catalog_manager.cc:4841] TS 131a1952b4fb46dc97f7737aa0b478ac (127.21.204.67:38157): tablet 2810224c964442f9ada5960ac67ec2f5 (table test-table [id=8947ee31eea74662ad503bcadf20d79b]) successfully deleted
I20260923 15:13:07.410897 1067 catalog_manager.cc:4841] TS 468be9f750bc4e9eb9057b40f7952b37 (127.21.204.66:37281): tablet 2810224c964442f9ada5960ac67ec2f5 (table test-table [id=8947ee31eea74662ad503bcadf20d79b]) successfully deleted
W20260923 15:13:07.411720 1068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:37061 (error 108)
I20260923 15:13:07.414018 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 1299
I20260923 15:13:07.418494 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:07.418330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37061 (local address 127.0.0.1:59820)
0923 15:13:07.418384 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:07.418443 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 15:13:07.418654 1068 catalog_manager.cc:4584] TS 3ccdd4ecc69b4eb484d7784ad1bc2e03 (127.21.204.65:37061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2810224c964442f9ada5960ac67ec2f5: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37061: connect: Connection refused (error 111)
I20260923 15:13:07.420435 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 1435
I20260923 15:13:07.426370 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 1049
I20260923 15:13:07.434882 22321 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 15:13:07.779419 22321 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 15:13:07.779539 22321 mini_ranger.cc:66] Stopping Ranger...
I20260923 15:13:08.124157 22321 mini_ranger.cc:68] Stopped Ranger
2026-09-23 15:13:08.124 UTC [634] LOG: received smart shutdown request
2026-09-23 15:13:08.127 UTC [634] LOG: background worker "logical replication launcher" (PID 643) exited with exit code 1
2026-09-23 15:13:08.128 UTC [638] LOG: shutting down
2026-09-23 15:13:08.315 UTC [634] LOG: database system is shut down
2026-09-23T15:13:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48375 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:13:08 dist-test-slave-t5lt krb5kdc[1619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:13:08 dist-test-slave-t5lt krb5kdc[1619](info): set up 2 sockets
Sep 23 15:13:08 dist-test-slave-t5lt krb5kdc[1619](info): commencing operation
krb5kdc: starting...
W20260923 15:13:09.455163 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:13:09 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:13:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/rangeradmin_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/rangeradmin_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/rangerlookup_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/rangerlookup_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
I20260923 15:13:09.620464 22321 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOKY8g3/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/postgres -l logfile start
2026-09-23 15:13:12.191 UTC [1649] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 15:13:12.192 UTC [1649] LOG: listening on IPv4 address "127.21.204.84", port 36723
2026-09-23 15:13:12.195 UTC [1649] LOG: listening on Unix socket "/tmp/.s.PGSQL.36723"
2026-09-23 15:13:12.217 UTC [1652] LOG: database system was shut down at 2026-09-23 15:13:10 UTC
2026-09-23 15:13:12.221 UTC [1649] LOG: database system is ready to accept connections
I20260923 15:13:13.183008 22321 mini_postgres.cc:96] Postgres bound to 36723
2026-09-23 15:13:13.189 UTC [1660] FATAL: database "slave" does not exist
127.21.204.84:36723 - accepting connections
I20260923 15:13:13.189849 22321 mini_ranger.cc:161] Starting Ranger...
I20260923 15:13:13.205269 22321 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 15:13:13.734217 22321 mini_ranger.cc:87] Created ranger Postgres database
I20260923 15:13:13.734306 22321 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-admin
2026-09-23 15:13:14,013 [I] DB FLAVOR :POSTGRES
2026-09-23 15:13:14,013 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 15:13:14,013 [I] Checking connection..
2026-09-23 15:13:14,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 15:13:14,265 [I] Checking connection passed.
2026-09-23 15:13:14,266 [I] --------- Verifying version history table ---------
2026-09-23 15:13:14,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 15:13:14,524 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 15:13:14,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 15:13:14,784 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 15:13:14,785 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 15:13:14,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 15:13:15.037 UTC [1724] WARNING: there is no transaction in progress
2026-09-23 15:13:15,044 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 15:13:15,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 15:13:15,289 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 15:13:15,290 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 15:13:15,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 15:13:15,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 15:13:15,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t5lt.c.gcp-upstream.internal','N') ;"
2026-09-23 15:13:16,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 15:13:16,278 [I] Table x_portal_user does not exist in database ranger
2026-09-23 15:13:16,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 15:13:16,531 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 15:13:16,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 15:13:16,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 15:13:17,052 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 15:13:17,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 15:13:18.071 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.087 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.105 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.122 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.138 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.155 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.208 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.217 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.238 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.251 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.264 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.281 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.292 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.300 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.308 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:18.316 UTC [1832] WARNING: there is no transaction in progress
2026-09-23 15:13:19,045 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 15:13:19,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 15:13:19,270 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 15:13:19,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 15:13:19,538 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 15:13:19,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 15:13:19,772 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 15:13:19,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 15:13:20,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 15:13:20,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t5lt.c.gcp-upstream.internal';"
2026-09-23 15:13:20,511 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 15:13:20,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 15:13:20,759 [I] DB_PATCHES have already been applied
I20260923 15:13:20.764967 22321 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOKY8g3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 15:13:20.765033 22321 mini_ranger.cc:193] Using host: 127.21.204.84
I20260923 15:13:20.768163 22321 mini_ranger.cc:227] Ranger admin URL: http://127.21.204.84:45987
Sep 23, 2026 3:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 3:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 3:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 3:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 3:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 3:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 3:13:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 15:13:21.876351 22321 mini_ranger.cc:160] Time spent starting Ranger: real 8.687s user 0.000s sys 0.009s
Sep 23, 2026 3:13:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 3:13:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 3:13:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 3:13:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 3:13:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 15:13:42.091512 22321 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.204.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.204.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/rangerkms_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.204.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/rangerkms_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.204.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.204.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.204.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/HTTP_127.21.204.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 15:13:42 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 15:13:42.174548 22321 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 15:13:42.183640 22321 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 15:13:43.128191 22321 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 15:13:43.128263 22321 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms
I20260923 15:13:43.128276 22321 mini_ranger_kms.cc:227] Using postgres at 127.21.204.84:36723
2026-09-23 15:13:43,297 [I] DB FLAVOR :POSTGRES
2026-09-23 15:13:43,298 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 15:13:43,298 [I] Checking connection
2026-09-23 15:13:43,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 15:13:43,543 [I] connection success
2026-09-23 15:13:43,544 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 15:13:43,544 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 15:13:43,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 15:13:43,769 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 15:13:43,770 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 15:13:43,770 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 15:13:43,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.204.84:36723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 15:13:44,044 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 15:13:44.483170 22321 mini_ranger_kms.cc:310] Created kms service
I20260923 15:13:44.699555 22321 mini_ranger_kms.cc:326] Created kudu user
I20260923 15:13:44.744715 22321 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 15:13:44 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:45.163497 22321 mini_ranger_kms.cc:384] Added ranger policy
I20260923 15:13:45.163702 22321 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOKY8g3/build/release/bin/postgresql.jar:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOKY8g3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 15:13:45.163724 22321 mini_ranger_kms.cc:240] Using host: 127.21.204.84
I20260923 15:13:45.167225 22321 mini_ranger_kms.cc:276] Ranger KMS PID: 2012
I20260923 15:13:45.167305 22321 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.204.84:55413
Sep 23, 2026 3:13:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 3:13:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 3:13:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 3:13:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 3:13:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 3:13:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 15:13:47 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790176427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.204.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:47.390471 22321 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.216s user 0.000s sys 0.152s
I20260923 15:13:47.390601 22321 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 15:13:47.390655 22321 mini_ranger_kms.cc:401] 127.21.204.84:55413/kms/v1/keys
Sep 23 15:13:47 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.204.84@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
Sep 23 15:13:48 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.204.84@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
Sep 23 15:13:50 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:13:50.284843 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34897
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34897
--ranger_config_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:50.386356 2063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:50.386559 2063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:50.386580 2063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:50.387748 2063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:13:50.387789 2063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:50.387804 2063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:50.387816 2063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:13:50.387828 2063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:13:50.389369 2063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.204.84:55413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34897
--ranger_config_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:50.389600 2063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:50.389863 2063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:50.395051 2071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:50.394989 2068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:50.395219 2069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:50.395112 2063 server_base.cc:1034] running on GCE node
I20260923 15:13:50.395475 2063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:50.395807 2063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:50.396994 2063 hybrid_clock.cc:648] HybridClock initialized: now 1790176430396967 us; error 40 us; skew 500 ppm
Sep 23 15:13:50 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:50.400069 2063 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:13:50.400497 2063 webserver.cc:460] Webserver started at http://127.21.204.126:37273/ using document root <none> and password file <none>
I20260923 15:13:50.400782 2063 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:50.400841 2063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:50.400980 2063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:50 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:50.465373 2063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "4b745a6222d449e7b96e626e8767a127"
format_stamp: "Formatted at 2026-09-23 15:13:50 on dist-test-slave-t5lt"
server_key: "6dd9deb82c81d3d416b78b9a43a31f66"
server_key_iv: "caf4f28f77172271133bdc5e01635df2"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:50.465905 2063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "4b745a6222d449e7b96e626e8767a127"
format_stamp: "Formatted at 2026-09-23 15:13:50 on dist-test-slave-t5lt"
server_key: "6dd9deb82c81d3d416b78b9a43a31f66"
server_key_iv: "caf4f28f77172271133bdc5e01635df2"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:50.467564 2063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:50.479549 2078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:50.479789 2063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:50.479905 2063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "4b745a6222d449e7b96e626e8767a127"
format_stamp: "Formatted at 2026-09-23 15:13:50 on dist-test-slave-t5lt"
server_key: "6dd9deb82c81d3d416b78b9a43a31f66"
server_key_iv: "caf4f28f77172271133bdc5e01635df2"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:50.480010 2063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:50.492288 2063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:50.494405 2063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:50.494568 2063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:50.499670 2063 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34897
I20260923 15:13:50.499738 2130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34897 every 8 connection(s)
I20260923 15:13:50.500135 2063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:13:50.500847 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:50.501300 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2063
I20260923 15:13:50.501408 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:13:50.501583 22321 external_mini_cluster.cc:1428] Setting key 47f3f49206abf9fe3c9da1b06989354c
I20260923 15:13:50.503898 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Bootstrap starting.
I20260923 15:13:50.504519 2131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:50.504802 2131 log.cc:824] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:50.505494 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: No bootstrap required, opened a new log
I20260923 15:13:50.507781 2131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:50.507977 2131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:50.508049 2131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b745a6222d449e7b96e626e8767a127, State: Initialized, Role: FOLLOWER
I20260923 15:13:50.508191 2131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:50.508292 2131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:13:50.508347 2131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:13:50.508402 2131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:50.508926 2131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:50.509038 2131 leader_election.cc:302] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b745a6222d449e7b96e626e8767a127; no voters:
I20260923 15:13:50.509243 2131 leader_election.cc:288] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:13:50.509335 2136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:13:50.509536 2136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 LEADER]: Becoming Leader. State: Replica: 4b745a6222d449e7b96e626e8767a127, State: Running, Role: LEADER
Sep 23 15:13:50 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:50.509699 2136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:50.509557 2131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:13:50.511550 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b745a6222d449e7b96e626e8767a127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } } }
I20260923 15:13:50.511638 2131 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 15:13:50.511628 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:50.511948 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b745a6222d449e7b96e626e8767a127. Latest consensus state: current_term: 1 leader_uuid: "4b745a6222d449e7b96e626e8767a127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } } }
I20260923 15:13:50.512032 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:50.512143 2150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:13:50.512609 2150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:13:50.514202 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:50.502324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.21.204.126:34897)
0923 15:13:50.502505 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:50.502515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:50.502547 (+ 32us) server_negotiation.cc:407] Connection header received
0923 15:13:50.503083 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:50.503092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:50.503152 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:50.503302 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:50.504063 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:50.505497 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:50.506633 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:50.507244 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:50.507646 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:50.510171 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:50.510175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:50.510176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:50.510187 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:50.512236 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:50.512654 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:50.512657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:50.512657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:50.512751 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:50.512987 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:50.512988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:50.512989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:50.513293 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:50.513442 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:50.513858 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 15:13:50.514005 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260923 15:13:50.514797 2150 catalog_manager.cc:1347] Generated new cluster ID: a9244169f80d4664a6f7b345300d5983
I20260923 15:13:50.514842 2150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:13:50.544086 2150 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:13:50.544955 2150 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:13:50.556331 2150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Generated new TSK 0
I20260923 15:13:50.556592 2150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:13:51.272913 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34897
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:51.386030 2176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:51.386282 2176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:51.386315 2176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:51.387583 2176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:51.387625 2176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:51.387651 2176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:13:51.389099 2176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.204.84:55413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:51.389320 2176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:51.389619 2176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:51.395740 2182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:51.395735 2184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:51.395735 2181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:51.396667 2176 server_base.cc:1034] running on GCE node
I20260923 15:13:51.397010 2176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:51.397279 2176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:51.398466 2176 hybrid_clock.cc:648] HybridClock initialized: now 1790176431398449 us; error 37 us; skew 500 ppm
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:51.401727 2176 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:13:51.402263 2176 webserver.cc:460] Webserver started at http://127.21.204.65:36325/ using document root <none> and password file <none>
I20260923 15:13:51.402544 2176 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:51.402604 2176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:51.402724 2176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:51.417850 2176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "9cb6fa093f5cd2f2a9ac97bd709cda60"
server_key_iv: "f993a38e3b76dfef4886430b82b40cf6"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.418419 2176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "9cb6fa093f5cd2f2a9ac97bd709cda60"
server_key_iv: "f993a38e3b76dfef4886430b82b40cf6"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.420272 2176 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260923 15:13:51.438042 2193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.438351 2176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 15:13:51.438481 2176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "9cb6fa093f5cd2f2a9ac97bd709cda60"
server_key_iv: "f993a38e3b76dfef4886430b82b40cf6"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.438597 2176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:51.462142 2176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:51.464457 2176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:51.464664 2176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:51.464936 2176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:51.465366 2176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:51.465457 2176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.465556 2176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:51.465631 2176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.478034 2176 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:43391
I20260923 15:13:51.478521 2176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:13:51.481036 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2176
I20260923 15:13:51.481140 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:13:51.481323 22321 external_mini_cluster.cc:1428] Setting key b69cd0231576f8d88386bd975ab6f04a
I20260923 15:13:51.486867 2306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:43391 every 8 connection(s)
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:51.511543 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:51.491083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:38073 (local address 127.21.204.126:34897)
0923 15:13:51.491327 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:51.491329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:51.494621 (+ 3292us) server_negotiation.cc:407] Connection header received
0923 15:13:51.494650 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:51.494651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:51.494665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:51.494768 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:51.497159 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.498021 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.499498 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.499825 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.500020 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:51.507314 (+ 7294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:51.507316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:51.507316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:51.507324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:51.509431 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:51.510095 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:51.510096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:51.510097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:51.510174 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:51.510553 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:51.510554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:51.510554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:51.510779 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:51.510853 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:51.511270 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 15:13:51.511406 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":144,"threads_started":1}
I20260923 15:13:51.512122 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:51.490692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34897 (local address 127.21.204.65:38073)
0923 15:13:51.493426 (+ 2734us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:51.493474 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:51.494468 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:51.496348 (+ 1880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:51.496353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:51.496771 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:51.497046 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:51.497052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.498124 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.498125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:51.499400 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:51.499403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.500184 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.500185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:51.500321 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:51.505137 (+ 4816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:51.505148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:51.507186 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:51.509592 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:51.509599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:51.509601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:51.509999 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:51.510325 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:51.510326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:51.510326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:51.510463 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:51.510899 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:51.510906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:51.511070 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 15:13:51.511721 (+ 651us) client_negotiation.cc:240] Negotiation successful
0923 15:13:51.511888 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2690,"thread_start_us":190,"threads_started":1}
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:13:51.512661 2307 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34897
I20260923 15:13:51.512755 2307 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:51.512946 2307 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:51.514359 2095 ts_manager.cc:194] Registered new tserver with Master: aa4e774a602c47b1bcf7e871e0fa5dd8 (127.21.204.65:43391)
I20260923 15:13:51.515307 2095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:38073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:13:51.532522 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34897
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 15:13:51.657366 2315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:51.657640 2315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:51.657675 2315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:51.659337 2315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:51.659425 2315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:51.659478 2315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:13:51.661528 2315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.204.84:55413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:51.661755 2315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:51.662045 2315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:51.668586 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:51.668701 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:51.668987 2315 server_base.cc:1034] running on GCE node
W20260923 15:13:51.669075 2323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:51.669397 2315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:51.669687 2315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:51.670864 2315 hybrid_clock.cc:648] HybridClock initialized: now 1790176431670839 us; error 50 us; skew 500 ppm
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:51.674002 2315 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:13:51.674520 2315 webserver.cc:460] Webserver started at http://127.21.204.66:41333/ using document root <none> and password file <none>
I20260923 15:13:51.674783 2315 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:51.674850 2315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:51.675028 2315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:51.687860 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "1cfedee610644b8a9babca97e4463054"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "5bf491dce0ff8951a5c0ea73350265b8"
server_key_iv: "d402a0539d84074a69aaf038d7381a59"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.688365 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "1cfedee610644b8a9babca97e4463054"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "5bf491dce0ff8951a5c0ea73350265b8"
server_key_iv: "d402a0539d84074a69aaf038d7381a59"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.694281 2315 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260923 15:13:51.708007 2331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.708245 2315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 15:13:51.708331 2315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "1cfedee610644b8a9babca97e4463054"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "5bf491dce0ff8951a5c0ea73350265b8"
server_key_iv: "d402a0539d84074a69aaf038d7381a59"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.708411 2315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:51.721161 2315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:51.723492 2315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:51.723654 2315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:51.723915 2315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:51.724274 2315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:51.724328 2315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.724396 2315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:51.724426 2315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.731839 2315 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:38565
I20260923 15:13:51.731943 2444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:38565 every 8 connection(s)
I20260923 15:13:51.732311 2315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:51.739195 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2315
I20260923 15:13:51.739310 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:13:51.739519 22321 external_mini_cluster.cc:1428] Setting key 71debbf6cad5a37b8feac0591f284f92
I20260923 15:13:51.741786 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:51.733071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:50547 (local address 127.21.204.126:34897)
0923 15:13:51.733123 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:51.733125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:51.733373 (+ 248us) server_negotiation.cc:407] Connection header received
0923 15:13:51.733916 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:51.733918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:51.733936 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:51.733996 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:51.734539 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.735045 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.736047 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.736267 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.736353 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:51.738453 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:51.738455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:51.738455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:51.738462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:51.740140 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:51.740645 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:51.740646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:51.740646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:51.740702 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:51.740977 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:51.740978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:51.740979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:51.741114 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:51.741155 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:51.741545 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 15:13:51.741650 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:13:51.742321 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:51.733076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34897 (local address 127.21.204.66:50547)
0923 15:13:51.733230 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:51.733270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:51.733777 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:51.734065 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:51.734069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:51.734256 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:51.734438 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:51.734443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.735145 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.735146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:51.735949 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:51.735951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:51.736377 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:51.736378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:51.736498 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:51.737005 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:51.737013 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:51.738357 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:51.740226 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:51.740233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:51.740235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:51.740553 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:51.740772 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:51.740772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:51.740773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:51.740906 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:51.741200 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:51.741205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:51.741370 (+ 165us) client_negotiation.cc:769] Sending connection context
0923 15:13:51.741895 (+ 525us) client_negotiation.cc:240] Negotiation successful
0923 15:13:51.742071 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260923 15:13:51.742908 2445 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34897
I20260923 15:13:51.743021 2445 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:51.743196 2445 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:51.743595 2095 ts_manager.cc:194] Registered new tserver with Master: 1cfedee610644b8a9babca97e4463054 (127.21.204.66:38565)
I20260923 15:13:51.744086 2095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:50547
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:13:51.768505 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34897
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:51.862092 2450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:51.862315 2450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:51.862354 2450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:51.863492 2450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:51.863559 2450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:51.863606 2450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:13:51.865330 2450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.204.84:55413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:51.865566 2450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:51.865823 2450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:51.872008 2458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:51.872104 2455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:51.872015 2456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:51.872388 2450 server_base.cc:1034] running on GCE node
I20260923 15:13:51.872777 2450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:51.873064 2450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:51.874214 2450 hybrid_clock.cc:648] HybridClock initialized: now 1790176431874188 us; error 42 us; skew 500 ppm
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:51.877449 2450 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:13:51.878060 2450 webserver.cc:460] Webserver started at http://127.21.204.67:46781/ using document root <none> and password file <none>
I20260923 15:13:51.878377 2450 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:51.878460 2450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:51.878604 2450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 15:13:51 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:51.895180 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "6788dea5efa2480a9e19706633858d5b"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "d8e057d824b3a2fd5425f8622c5318b2"
server_key_iv: "213d899abeb59cd4686f9103be5913b9"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.895661 2450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "6788dea5efa2480a9e19706633858d5b"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "d8e057d824b3a2fd5425f8622c5318b2"
server_key_iv: "213d899abeb59cd4686f9103be5913b9"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.897625 2450 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260923 15:13:51.914949 2465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.918586 2450 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260923 15:13:51.918735 2450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "6788dea5efa2480a9e19706633858d5b"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "d8e057d824b3a2fd5425f8622c5318b2"
server_key_iv: "213d899abeb59cd4686f9103be5913b9"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:51.918846 2450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:51.986392 2450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:51.988831 2450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:51.988986 2450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:51.989279 2450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:51.989688 2450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:51.989753 2450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.989794 2450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:51.989817 2450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:51.998162 2450 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:38889
I20260923 15:13:51.998651 2450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:13:52.000913 2578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:38889 every 8 connection(s)
I20260923 15:13:52.006812 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2450
I20260923 15:13:52.006929 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:13:52.007124 22321 external_mini_cluster.cc:1428] Setting key f2ca7df20e9988d77e0fd24806793298
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176431, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:52.012935 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.000206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:48201 (local address 127.21.204.126:34897)
0923 15:13:52.000258 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:52.000261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:52.000377 (+ 116us) server_negotiation.cc:407] Connection header received
0923 15:13:52.001273 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:52.001275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:52.001290 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:52.001373 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:52.002463 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.003242 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.004775 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.005083 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.005167 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.009304 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:52.009305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:52.009305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:52.009313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:52.011230 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.011765 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.011766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.011766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.011829 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.012134 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.012135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.012135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.012277 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:52.012323 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:52.012740 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 15:13:52.012833 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 15:13:52.013466 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:51.999737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34897 (local address 127.21.204.67:48201)
0923 15:13:52.000093 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:52.000148 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:52.000976 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:52.001541 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:52.001550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:52.002034 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:52.002341 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.002348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.003365 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.003366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.004672 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.004675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.005205 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.005207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.005356 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.005996 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:52.006007 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:52.009190 (+ 3183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:52.011330 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.011337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.011340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.011669 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.011904 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.011905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.011905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.012044 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.012377 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:52.012383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:52.012553 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 15:13:52.013065 (+ 512us) client_negotiation.cc:240] Negotiation successful
0923 15:13:52.013230 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":228,"threads_started":1}
I20260923 15:13:52.013921 2579 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34897
I20260923 15:13:52.014017 2579 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.014207 2579 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:52.014550 2095 ts_manager.cc:194] Registered new tserver with Master: 6788dea5efa2480a9e19706633858d5b (127.21.204.67:38889)
I20260923 15:13:52.015045 2095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:48201
I20260923 15:13:52.018985 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:13:52.019694 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2176
I20260923 15:13:52.027369 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2315
I20260923 15:13:52.032968 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2450
I20260923 15:13:52.038093 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2063
I20260923 15:13:52.043745 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34897
--webserver_interface=127.21.204.126
--webserver_port=37273
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34897
--ranger_config_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:52.123920 2582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:52.124123 2582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:52.124152 2582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:52.125280 2582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:13:52.125324 2582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:52.125337 2582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:52.125350 2582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:13:52.125362 2582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:13:52.126987 2582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34897
--ranger_config_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=37273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:52.127252 2582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:52.127470 2582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:52.132226 2590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:52.132308 2588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:52.132262 2587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:52.132440 2582 server_base.cc:1034] running on GCE node
I20260923 15:13:52.132668 2582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:52.132946 2582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:52.134084 2582 hybrid_clock.cc:648] HybridClock initialized: now 1790176432134073 us; error 29 us; skew 500 ppm
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:52.136529 2582 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:13:52.136976 2582 webserver.cc:460] Webserver started at http://127.21.204.126:37273/ using document root <none> and password file <none>
I20260923 15:13:52.137208 2582 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:52.137281 2582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:52.193542 2582 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:52.194463 2598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.194684 2582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:13:52.194789 2582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "4b745a6222d449e7b96e626e8767a127"
format_stamp: "Formatted at 2026-09-23 15:13:50 on dist-test-slave-t5lt"
server_key: "6dd9deb82c81d3d416b78b9a43a31f66"
server_key_iv: "caf4f28f77172271133bdc5e01635df2"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:52.195259 2582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:52.210672 2582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:52.212707 2582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:52.212865 2582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:52.217027 2582 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34897
I20260923 15:13:52.217149 2650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34897 every 8 connection(s)
I20260923 15:13:52.217458 2582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:13:52.218760 2651 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:13:52.219249 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2582
I20260923 15:13:52.219441 2651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Bootstrap starting.
I20260923 15:13:52.219530 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:43391
--local_ip_for_outbound_sockets=127.21.204.65
--tserver_master_addrs=127.21.204.126:34897
--webserver_port=36325
--webserver_interface=127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:13:52.229835 2651 log.cc:824] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:52.232091 2651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:13:52.232394 2651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Bootstrap complete.
I20260923 15:13:52.235672 2651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:52.235981 2651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b745a6222d449e7b96e626e8767a127, State: Initialized, Role: FOLLOWER
I20260923 15:13:52.236138 2651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:52.236203 2651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:13:52.236240 2651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:13:52.236284 2651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 1 FOLLOWER]: Advancing to term 2
I20260923 15:13:52.237550 2651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:52.237656 2651 leader_election.cc:302] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b745a6222d449e7b96e626e8767a127; no voters:
I20260923 15:13:52.237811 2651 leader_election.cc:288] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 15:13:52.237903 2655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 15:13:52.238070 2651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:13:52.238076 2655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [term 2 LEADER]: Becoming Leader. State: Replica: 4b745a6222d449e7b96e626e8767a127, State: Running, Role: LEADER
I20260923 15:13:52.238277 2655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } }
I20260923 15:13:52.238593 2656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b745a6222d449e7b96e626e8767a127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } } }
I20260923 15:13:52.238593 2655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b745a6222d449e7b96e626e8767a127. Latest consensus state: current_term: 2 leader_uuid: "4b745a6222d449e7b96e626e8767a127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b745a6222d449e7b96e626e8767a127" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34897 } } }
I20260923 15:13:52.238694 2656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:52.238698 2655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:52.239231 2664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:13:52.239503 2651 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 15:13:52.239849 2664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:13:52.242345 2664 catalog_manager.cc:1259] Loaded cluster ID: a9244169f80d4664a6f7b345300d5983
I20260923 15:13:52.242460 2664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:13:52.244196 2664 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:13:52.244385 2664 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Loaded TSK: 0
I20260923 15:13:52.244609 2664 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 15:13:52.335330 2653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:52.335608 2653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:52.335651 2653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:52.337343 2653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:52.337396 2653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:52.337433 2653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:13:52.339665 2653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=36325
--tserver_master_addrs=127.21.204.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:52.339906 2653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:52.340161 2653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:52.346701 2688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:52.346710 2689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:52.346999 2691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:52.347496 2653 server_base.cc:1034] running on GCE node
I20260923 15:13:52.347708 2653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:52.347991 2653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:52.349164 2653 hybrid_clock.cc:648] HybridClock initialized: now 1790176432349139 us; error 39 us; skew 500 ppm
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:52.352469 2653 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:13:52.353066 2653 webserver.cc:460] Webserver started at http://127.21.204.65:36325/ using document root <none> and password file <none>
I20260923 15:13:52.353317 2653 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:52.353386 2653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:52.390386 2653 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260923 15:13:52.391505 2699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.391711 2653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:13:52.391798 2653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "9cb6fa093f5cd2f2a9ac97bd709cda60"
server_key_iv: "f993a38e3b76dfef4886430b82b40cf6"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:52.392258 2653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:52.419380 2653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:52.422055 2653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:52.422269 2653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:52.422550 2653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:52.422998 2653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:52.423110 2653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.423218 2653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:52.423295 2653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.430932 2653 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:43391
I20260923 15:13:52.431411 2653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:13:52.431852 2812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:43391 every 8 connection(s)
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:52.441094 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2653
I20260923 15:13:52.441421 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:38565
--local_ip_for_outbound_sockets=127.21.204.66
--tserver_master_addrs=127.21.204.126:34897
--webserver_port=41333
--webserver_interface=127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:13:52.446947 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.432568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34897 (local address 127.21.204.65:37987)
0923 15:13:52.432957 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:52.433003 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:52.433657 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:52.434231 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:52.434235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:52.434467 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:52.434735 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.434741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.436424 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.436425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.437782 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.437785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.438738 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.438739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.438862 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.439476 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:52.439486 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:52.442238 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:52.444550 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.444558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.444560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.444852 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.445523 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.445524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.445524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.445662 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.446234 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:52.446240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:52.446394 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 15:13:52.446511 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 15:13:52.446679 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":97,"threads_started":1}
I20260923 15:13:52.457259 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.432659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:37987 (local address 127.21.204.126:34897)
0923 15:13:52.432866 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:52.432875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:52.433184 (+ 309us) server_negotiation.cc:407] Connection header received
0923 15:13:52.433950 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:52.433957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:52.434019 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:52.434170 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:52.434872 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.436284 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.437918 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.438608 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.438733 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.442370 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:52.442375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:52.442376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:52.442391 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:52.444435 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.445303 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.445307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.445307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.445429 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.445742 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.445743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.445743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.446108 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:52.446204 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:52.456886 (+ 10682us) server_negotiation.cc:299] Negotiation successful
0923 15:13:52.457049 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260923 15:13:52.458019 2813 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34897
I20260923 15:13:52.458123 2813 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.458329 2813 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
W20260923 15:13:52.459322 2813 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:34897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 15:13:52.459393 2813 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.459455 2813 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:52.459870 2813 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.459946 2813 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
W20260923 15:13:52.460275 2813 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 15:13:52.547345 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:52.547544 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:52.547564 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:52.548703 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:52.548744 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:52.548770 2817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:13:52.550282 2817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:38565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=41333
--tserver_master_addrs=127.21.204.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:52.550524 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:52.550781 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:52.556113 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:52.556116 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:52.556324 2817 server_base.cc:1034] running on GCE node
W20260923 15:13:52.556116 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:52.556599 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:52.556835 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:52.558001 2817 hybrid_clock.cc:648] HybridClock initialized: now 1790176432557984 us; error 31 us; skew 500 ppm
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:52.560757 2817 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:13:52.561169 2817 webserver.cc:460] Webserver started at http://127.21.204.66:41333/ using document root <none> and password file <none>
I20260923 15:13:52.561400 2817 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:52.561483 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:52.582293 2817 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260923 15:13:52.583406 2833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.583617 2817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:13:52.583702 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "1cfedee610644b8a9babca97e4463054"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "5bf491dce0ff8951a5c0ea73350265b8"
server_key_iv: "d402a0539d84074a69aaf038d7381a59"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:52.584137 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:52.601940 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:52.604386 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:52.604599 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:52.604861 2817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:52.605258 2817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:52.605350 2817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.605414 2817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:52.605463 2817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.612051 2817 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:38565
I20260923 15:13:52.612550 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:13:52.619035 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2817
I20260923 15:13:52.619328 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.204.84:55413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:38889
--local_ip_for_outbound_sockets=127.21.204.67
--tserver_master_addrs=127.21.204.126:34897
--webserver_port=46781
--webserver_interface=127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:52.622771 2946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:38565 every 8 connection(s)
I20260923 15:13:52.626183 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.613710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:36731 (local address 127.21.204.126:34897)
0923 15:13:52.613758 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:52.613760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:52.614055 (+ 295us) server_negotiation.cc:407] Connection header received
0923 15:13:52.615252 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:52.615253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:52.615269 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:52.615300 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:52.616139 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.616923 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.619748 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.620038 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.620089 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.622742 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:52.622743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:52.622744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:52.622752 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:52.624529 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.625033 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.625034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.625035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.625094 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.625394 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.625395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.625395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.625527 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:52.625598 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:52.625989 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 15:13:52.626082 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:13:52.626930 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.613700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34897 (local address 127.21.204.66:36731)
0923 15:13:52.613913 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:52.613964 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:52.615118 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:52.615481 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:52.615485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:52.615739 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:52.616016 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.616023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.618011 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.618012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.619649 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.619651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.620210 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.620211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.620350 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.620927 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:52.620937 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:52.622627 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:52.624641 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.624647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.624649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.624941 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.625169 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.625170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.625171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.625304 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.625635 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:52.625642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:52.625804 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 15:13:52.626522 (+ 718us) client_negotiation.cc:240] Negotiation successful
0923 15:13:52.626700 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260923 15:13:52.627624 2947 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34897
I20260923 15:13:52.627717 2947 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.627907 2947 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
W20260923 15:13:52.628589 2947 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:34897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 15:13:52.628654 2947 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.628712 2947 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:52.629021 2947 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.629097 2947 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
W20260923 15:13:52.629413 2947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 15:13:52.751093 2950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:52.751478 2950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:52.751585 2950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:52.753345 2950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:52.753572 2950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:52.753729 2950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:13:52.756034 2950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.204.84:55413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:38889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=46781
--tserver_master_addrs=127.21.204.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:52.756511 2950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:52.756876 2950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:52.769129 2959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:52.769131 2956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:52.772888 2950 server_base.cc:1034] running on GCE node
W20260923 15:13:52.774428 2957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:52.774747 2950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:52.775043 2950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:52.776216 2950 hybrid_clock.cc:648] HybridClock initialized: now 1790176432776194 us; error 34 us; skew 500 ppm
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:52.779520 2950 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:13:52.780038 2950 webserver.cc:460] Webserver started at http://127.21.204.67:46781/ using document root <none> and password file <none>
I20260923 15:13:52.780313 2950 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:52.780467 2950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:52.797084 2950 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:52.806397 2967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.806639 2950 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.000s
I20260923 15:13:52.806739 2950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "6788dea5efa2480a9e19706633858d5b"
format_stamp: "Formatted at 2026-09-23 15:13:51 on dist-test-slave-t5lt"
server_key: "d8e057d824b3a2fd5425f8622c5318b2"
server_key_iv: "213d899abeb59cd4686f9103be5913b9"
server_key_version: "kuduclusterkey@0"
I20260923 15:13:52.807256 2950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:52.830494 2950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:52.832763 2950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:52.832926 2950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:52.833211 2950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:13:52.833586 2950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:13:52.833639 2950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.833679 2950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:13:52.833705 2950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:52.840464 2950 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:38889
I20260923 15:13:52.840932 2950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:13:52.842607 3080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:38889 every 8 connection(s)
I20260923 15:13:52.846436 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 2950
Sep 23 15:13:52 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176432, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:52.854044 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.843527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:45665 (local address 127.21.204.126:34897)
0923 15:13:52.843602 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:52.843604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:52.844016 (+ 412us) server_negotiation.cc:407] Connection header received
0923 15:13:52.844649 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:52.844651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:52.844665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:52.844744 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:52.845430 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.846226 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.847733 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.848012 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.848062 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.850722 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:52.850723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:52.850724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:52.850732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:52.852397 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.852901 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.852902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.852903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.852965 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.853262 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.853263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.853263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.853391 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:52.853433 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:52.853808 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 15:13:52.853937 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 15:13:52.854806 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.843290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34897 (local address 127.21.204.67:45665)
0923 15:13:52.843865 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:52.843913 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:52.844511 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:52.844836 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:52.844840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:52.845055 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:52.845324 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.845329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.846342 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.846343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.847622 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:52.847626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.848191 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.848192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:52.848340 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.848915 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:52.848925 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:52.850606 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:52.852491 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.852498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.852501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.852799 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.853039 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:52.853039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:52.853040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:52.853181 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:52.853479 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:52.853485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:52.853635 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 15:13:52.854420 (+ 785us) client_negotiation.cc:240] Negotiation successful
0923 15:13:52.854580 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":96,"threads_started":1}
I20260923 15:13:52.855458 3081 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34897
I20260923 15:13:52.855549 3081 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.855726 3081 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
W20260923 15:13:52.856407 3081 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:34897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 15:13:52.856479 3081 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.856545 3081 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:52.856915 3081 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:52.856989 3081 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
W20260923 15:13:52.857349 3081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 15:13:52.872736 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:52.846977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.21.204.126:34897)
0923 15:13:52.847238 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:52.847240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:52.847247 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:13:52.847266 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:52.847267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:52.847279 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:52.847334 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:52.851287 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.852275 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.853552 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:52.853823 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:52.853897 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:52.855811 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:52.855812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:52.855812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:52.855819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:52.871173 (+ 15354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.871648 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.871649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.871649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.871727 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:52.871944 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:52.871946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:52.871947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:52.872116 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:52.872320 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:52.872517 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 15:13:52.872625 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":50,"threads_started":1}
Sep 23 15:13:53 dist-test-slave-t5lt krb5kdc[1619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790176433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:53.460465 2813 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:53.460621 2813 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:53.461030 2615 ts_manager.cc:194] Registered new tserver with Master: aa4e774a602c47b1bcf7e871e0fa5dd8 (127.21.204.65:43391)
I20260923 15:13:53.462080 2615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:37987
I20260923 15:13:53.629594 2947 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:53.629752 2947 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:53.630084 2615 ts_manager.cc:194] Registered new tserver with Master: 1cfedee610644b8a9babca97e4463054 (127.21.204.66:38565)
I20260923 15:13:53.630611 2615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:36731
Sep 23 15:13:53 dist-test-slave-t5lt krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176433, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for HTTP/127.21.204.84@KRBTEST.COM
I20260923 15:13:53.857537 3081 heartbeater.cc:464] Registering TS with master...
I20260923 15:13:53.857698 3081 heartbeater.cc:511] Master 127.21.204.126:34897 requested a full tablet report, sending...
I20260923 15:13:53.858073 2615 ts_manager.cc:194] Registered new tserver with Master: 6788dea5efa2480a9e19706633858d5b (127.21.204.67:38889)
I20260923 15:13:53.858603 2615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:45665
I20260923 15:13:53.862191 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:13:53.871150 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.863229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.21.204.126:34897)
0923 15:13:53.863499 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:53.863501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:53.863662 (+ 161us) server_negotiation.cc:407] Connection header received
0923 15:13:53.863767 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:53.863768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:53.863781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:53.863826 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:53.864541 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.865270 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.866355 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.866662 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.866707 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.868195 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:53.868196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:53.868197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:53.868204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:53.869860 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:53.870258 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:53.870259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:53.870259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:53.870315 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:53.870524 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:53.870526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:53.870526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:53.870667 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:53.870747 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:53.870963 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 15:13:53.871048 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":67,"threads_started":1}
I20260923 15:13:53.873669 2615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:13:53.876068 2615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:13:53.897943 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.883410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:38889 (local address 127.0.0.1:60118)
0923 15:13:53.884171 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:53.884211 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:53.884275 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:53.884821 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:53.884825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:53.884828 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:53.884894 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.884896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.892300 (+ 7404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.892301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.893346 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.893348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.897619 (+ 4271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.897620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.897702 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.897715 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 15:13:53.897783 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 15:13:53.897839 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":80,"threads_started":1}
I20260923 15:13:53.898897 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.883743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.21.204.67:38889)
0923 15:13:53.884577 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:53.884580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:53.884591 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:13:53.884626 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:53.884628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:53.884646 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:53.884727 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:53.884983 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.886409 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.894011 (+ 7602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.895304 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.898663 (+ 3359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.898733 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:53.898760 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 15:13:53.898819 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":43,"threads_started":1}
I20260923 15:13:53.899384 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.883413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:38565 (local address 127.0.0.1:34012)
0923 15:13:53.886527 (+ 3114us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:53.886537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:53.886585 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:53.888841 (+ 2256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:53.888842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:53.888844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:53.888893 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.888895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.891022 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.891023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.892175 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.892179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.898580 (+ 6401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.898581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.899185 (+ 604us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.899191 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 15:13:53.899267 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 15:13:53.899305 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3095,"thread_start_us":87,"threads_started":1}
I20260923 15:13:53.899808 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.883866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.21.204.66:38565)
0923 15:13:53.884509 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:53.884513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:53.886649 (+ 2136us) server_negotiation.cc:407] Connection header received
0923 15:13:53.886687 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:53.886689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:53.886704 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:53.886775 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:53.888974 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.890029 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.893427 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.897431 (+ 4004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.897501 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.897553 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:53.899650 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0923 15:13:53.899731 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":56,"threads_started":1}
I20260923 15:13:53.900502 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.883998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.21.204.65:43391)
0923 15:13:53.884438 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:53.884443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:53.887000 (+ 2557us) server_negotiation.cc:407] Connection header received
0923 15:13:53.887038 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:53.887040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:53.887056 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:53.887143 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:53.887351 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.888754 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.896160 (+ 7406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.896939 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.897022 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.897099 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:53.900358 (+ 3259us) server_negotiation.cc:299] Negotiation successful
0923 15:13:53.900424 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":57,"threads_started":1}
I20260923 15:13:53.900290 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.883673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:43391 (local address 127.0.0.1:60318)
0923 15:13:53.886886 (+ 3213us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:53.886896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:53.886944 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:53.887227 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:53.887228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:53.887230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:53.887269 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.887270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.890138 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.890139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.896045 (+ 5906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.896049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.900057 (+ 4008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.900058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.900130 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.900134 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:13:53.900189 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 15:13:53.900225 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3196,"thread_start_us":79,"threads_started":1}
I20260923 15:13:53.901598 2871 tablet_service.cc:1467] Processing CreateTablet for tablet f60080973a20415ca15998dc9d2a25e6 (DEFAULT_TABLE table=test-table [id=fae07ca2de78461daf754a464f5e0db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:13:53.901587 2996 tablet_service.cc:1467] Processing CreateTablet for tablet f60080973a20415ca15998dc9d2a25e6 (DEFAULT_TABLE table=test-table [id=fae07ca2de78461daf754a464f5e0db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:13:53.902019 2996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60080973a20415ca15998dc9d2a25e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:53.902117 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60080973a20415ca15998dc9d2a25e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:53.906066 2747 tablet_service.cc:1467] Processing CreateTablet for tablet f60080973a20415ca15998dc9d2a25e6 (DEFAULT_TABLE table=test-table [id=fae07ca2de78461daf754a464f5e0db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:13:53.906400 2747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60080973a20415ca15998dc9d2a25e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:53.918067 3114 tablet_bootstrap.cc:492] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Bootstrap starting.
I20260923 15:13:53.918880 3114 tablet_bootstrap.cc:654] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:53.919212 3114 log.cc:824] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:53.921830 3116 tablet_bootstrap.cc:492] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8: Bootstrap starting.
I20260923 15:13:53.922688 3116 tablet_bootstrap.cc:654] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:53.923223 3116 log.cc:824] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:53.923741 3115 tablet_bootstrap.cc:492] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Bootstrap starting.
I20260923 15:13:53.924854 3115 tablet_bootstrap.cc:654] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:53.925230 3115 log.cc:824] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:53.926193 3114 tablet_bootstrap.cc:492] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: No bootstrap required, opened a new log
I20260923 15:13:53.926321 3114 ts_tablet_manager.cc:1397] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260923 15:13:53.928682 3114 raft_consensus.cc:348] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.928984 3114 raft_consensus.cc:374] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:53.929070 3114 raft_consensus.cc:729] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfedee610644b8a9babca97e4463054, State: Initialized, Role: FOLLOWER
I20260923 15:13:53.929270 3114 consensus_queue.cc:260] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.929612 3114 ts_tablet_manager.cc:1428] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:13:53.929795 2947 heartbeater.cc:503] Master 127.21.204.126:34897 was elected leader, sending a full tablet report...
I20260923 15:13:53.931737 3116 tablet_bootstrap.cc:492] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8: No bootstrap required, opened a new log
I20260923 15:13:53.931864 3116 ts_tablet_manager.cc:1397] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260923 15:13:53.934571 3116 raft_consensus.cc:348] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.934796 3116 raft_consensus.cc:374] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:53.934882 3116 raft_consensus.cc:729] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4e774a602c47b1bcf7e871e0fa5dd8, State: Initialized, Role: FOLLOWER
I20260923 15:13:53.935109 3116 consensus_queue.cc:260] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.935427 3116 ts_tablet_manager.cc:1428] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:13:53.935592 2813 heartbeater.cc:503] Master 127.21.204.126:34897 was elected leader, sending a full tablet report...
I20260923 15:13:53.936960 3115 tablet_bootstrap.cc:492] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: No bootstrap required, opened a new log
I20260923 15:13:53.937038 3115 ts_tablet_manager.cc:1397] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260923 15:13:53.939910 3115 raft_consensus.cc:348] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.940084 3115 raft_consensus.cc:374] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:53.940122 3115 raft_consensus.cc:729] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6788dea5efa2480a9e19706633858d5b, State: Initialized, Role: FOLLOWER
I20260923 15:13:53.940848 3115 consensus_queue.cc:260] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.941224 3115 ts_tablet_manager.cc:1428] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 15:13:53.941385 3081 heartbeater.cc:503] Master 127.21.204.126:34897 was elected leader, sending a full tablet report...
I20260923 15:13:53.974402 3121 raft_consensus.cc:482] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:13:53.974557 3121 raft_consensus.cc:504] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.974977 3121 leader_election.cc:288] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfedee610644b8a9babca97e4463054 (127.21.204.66:38565), 6788dea5efa2480a9e19706633858d5b (127.21.204.67:38889)
I20260923 15:13:53.985498 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.975871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:58285 (local address 127.21.204.66:38565)
0923 15:13:53.976013 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:53.976016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:53.979102 (+ 3086us) server_negotiation.cc:407] Connection header received
0923 15:13:53.979231 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:53.979232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:53.979245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:53.979279 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:53.979543 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.980282 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.984496 (+ 4214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.985049 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.985107 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.985118 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:53.985372 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 15:13:53.985417 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260923 15:13:53.986073 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.975274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:38565 (local address 127.21.204.65:58285)
0923 15:13:53.979015 (+ 3741us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:53.979030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:53.979140 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:53.979359 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:53.979360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:53.979362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:53.979452 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.979454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.980403 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.980405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.984400 (+ 3995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.984402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.985206 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.985207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.985278 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.985282 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:13:53.985864 (+ 582us) client_negotiation.cc:240] Negotiation successful
0923 15:13:53.985959 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3706,"thread_start_us":115,"threads_started":1}
I20260923 15:13:53.987494 2901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60080973a20415ca15998dc9d2a25e6" candidate_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfedee610644b8a9babca97e4463054" is_pre_election: true
I20260923 15:13:53.987677 2901 raft_consensus.cc:2454] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4e774a602c47b1bcf7e871e0fa5dd8 in term 0.
I20260923 15:13:53.988071 2700 leader_election.cc:302] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfedee610644b8a9babca97e4463054, aa4e774a602c47b1bcf7e871e0fa5dd8; no voters:
I20260923 15:13:53.988214 3121 raft_consensus.cc:2781] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:13:53.988257 3121 raft_consensus.cc:482] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:13:53.988283 3121 raft_consensus.cc:3037] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:53.989336 3121 raft_consensus.cc:504] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.989482 3121 leader_election.cc:288] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [CANDIDATE]: Term 1 election: Requested vote from peers 1cfedee610644b8a9babca97e4463054 (127.21.204.66:38565), 6788dea5efa2480a9e19706633858d5b (127.21.204.67:38889)
I20260923 15:13:53.989955 2901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60080973a20415ca15998dc9d2a25e6" candidate_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfedee610644b8a9babca97e4463054"
I20260923 15:13:53.990046 2901 raft_consensus.cc:3037] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:53.990903 2901 raft_consensus.cc:2454] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4e774a602c47b1bcf7e871e0fa5dd8 in term 1.
I20260923 15:13:53.991161 2700 leader_election.cc:302] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfedee610644b8a9babca97e4463054, aa4e774a602c47b1bcf7e871e0fa5dd8; no voters:
I20260923 15:13:53.991290 3121 raft_consensus.cc:2781] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:13:53.991446 3121 raft_consensus.cc:686] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [term 1 LEADER]: Becoming Leader. State: Replica: aa4e774a602c47b1bcf7e871e0fa5dd8, State: Running, Role: LEADER
I20260923 15:13:53.991534 3121 consensus_queue.cc:237] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } }
I20260923 15:13:53.992427 2615 catalog_manager.cc:5495] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa4e774a602c47b1bcf7e871e0fa5dd8 (127.21.204.65). New cstate: current_term: 1 leader_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 43391 } health_report { overall_health: HEALTHY } } }
I20260923 15:13:53.993132 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.975958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:51529 (local address 127.21.204.67:38889)
0923 15:13:53.976051 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:53.976053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:53.981738 (+ 5685us) server_negotiation.cc:407] Connection header received
0923 15:13:53.981858 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:53.981859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:53.981889 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:53.981935 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:13:53.982157 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.982854 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.986337 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.986904 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.986976 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.986984 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:53.993003 (+ 6019us) server_negotiation.cc:299] Negotiation successful
0923 15:13:53.993063 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 15:13:53.993758 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:53.975534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:38889 (local address 127.21.204.65:51529)
0923 15:13:53.981661 (+ 6127us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:53.981673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:53.981769 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:53.982007 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:53.982008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:53.982010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:13:53.982077 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.982078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.982985 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.982986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.984190 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:53.984192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:53.987052 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:53.987053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:53.987129 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:53.987132 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:13:53.993628 (+ 6496us) client_negotiation.cc:240] Negotiation successful
0923 15:13:53.993679 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6101,"thread_start_us":71,"threads_started":1}
I20260923 15:13:53.994316 3027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60080973a20415ca15998dc9d2a25e6" candidate_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6788dea5efa2480a9e19706633858d5b" is_pre_election: true
I20260923 15:13:53.994467 3027 raft_consensus.cc:2454] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa4e774a602c47b1bcf7e871e0fa5dd8 in term 0.
I20260923 15:13:53.994575 3034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60080973a20415ca15998dc9d2a25e6" candidate_uuid: "aa4e774a602c47b1bcf7e871e0fa5dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6788dea5efa2480a9e19706633858d5b"
I20260923 15:13:53.994625 3034 raft_consensus.cc:3037] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:53.995486 3034 raft_consensus.cc:2454] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa4e774a602c47b1bcf7e871e0fa5dd8 in term 1.
I20260923 15:13:54.006381 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:54.002702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.21.204.65:43391)
0923 15:13:54.002756 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:54.002759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:54.002795 (+ 36us) server_negotiation.cc:407] Connection header received
0923 15:13:54.002873 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:54.002874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:54.002896 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:54.002958 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:13:54.003136 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:54.003854 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:54.004993 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:54.005298 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:54.005361 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:54.005799 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:13:54.005958 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:13:54.006056 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:54.006279 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 15:13:54.006302 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:13:54.008646 3034 raft_consensus.cc:1261] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 1 FOLLOWER]: Refusing update from remote peer aa4e774a602c47b1bcf7e871e0fa5dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:13:54.009085 3125 consensus_queue.cc:1035] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6788dea5efa2480a9e19706633858d5b" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:13:54.011574 2901 raft_consensus.cc:1261] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 1 FOLLOWER]: Refusing update from remote peer aa4e774a602c47b1bcf7e871e0fa5dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:13:54.011978 3125 consensus_queue.cc:1035] T f60080973a20415ca15998dc9d2a25e6 P aa4e774a602c47b1bcf7e871e0fa5dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfedee610644b8a9babca97e4463054" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:13:54.016729 3128 mvcc.cc:205] Tried to move back new op lower bound from 7332562673696313344 to 7332562673629904896. Current Snapshot: MvccSnapshot[applied={T|T < 7332562673696313344}]
I20260923 15:13:54.016955 3131 mvcc.cc:205] Tried to move back new op lower bound from 7332562673696313344 to 7332562673629904896. Current Snapshot: MvccSnapshot[applied={T|T < 7332562673696313344}]
I20260923 15:13:54.019703 2615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:13:54.019794 2615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:13:54.020603 2615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b745a6222d449e7b96e626e8767a127: Sending DeleteTablet for 3 replicas of tablet f60080973a20415ca15998dc9d2a25e6
I20260923 15:13:54.021019 2871 tablet_service.cc:1514] Processing DeleteTablet for tablet f60080973a20415ca15998dc9d2a25e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:13:54 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:34012
I20260923 15:13:54.021250 2996 tablet_service.cc:1514] Processing DeleteTablet for tablet f60080973a20415ca15998dc9d2a25e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:13:54 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:60118
I20260923 15:13:54.021677 2747 tablet_service.cc:1514] Processing DeleteTablet for tablet f60080973a20415ca15998dc9d2a25e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:13:54 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:60318
I20260923 15:13:54.022251 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2653
I20260923 15:13:54.023469 3136 tablet_replica.cc:330] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: stopping tablet replica
I20260923 15:13:54.023587 3136 raft_consensus.cc:2229] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:13:54.023705 3136 raft_consensus.cc:2258] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:13:54.023860 3137 tablet_replica.cc:330] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: stopping tablet replica
I20260923 15:13:54.023963 3137 raft_consensus.cc:2229] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:13:54.024039 3137 pending_rounds.cc:62] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Trying to abort 2 pending ops.
I20260923 15:13:54.024075 3137 pending_rounds.cc:68] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332562673696313344 op_type: WRITE_OP write_request { tablet_id: "f60080973a20415ca15998dc9d2a25e6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\336\333\317\325\006\"8\n\ntest-admin\022*\n fae07ca2de78461daf754a464f5e0db2\020\001\030\001 \001(\001" signature: "<redacted>""\035D\372\337\305\233\217\300\365\327\312\324\331(\217\313\323\252UG\240\217\352\335\210\215\345\342\250\017\3058\023\210\260\352\005\261\r\342e\006X\344Z\247^\223\313\231!E\200\336\026c\222\240Ua\005\271K\247" signing_key_seq_num: 0 } } request_id { client_id: "3856af2190c8457a9920f9cc360aee25" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260923 15:13:54.024302 3137 pending_rounds.cc:68] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332562673629904896 op_type: NO_OP noop_request { }
I20260923 15:13:54.024360 3137 raft_consensus.cc:2866] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260923 15:13:54.024402 3137 raft_consensus.cc:2258] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:13:54.024847 3137 ts_tablet_manager.cc:1905] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:13:54.024989 3136 ts_tablet_manager.cc:1905] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:13:54.026875 3137 ts_tablet_manager.cc:1918] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:13:54.026954 3137 log.cc:1196] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/f60080973a20415ca15998dc9d2a25e6
I20260923 15:13:54.027213 3137 ts_tablet_manager.cc:1939] T f60080973a20415ca15998dc9d2a25e6 P 6788dea5efa2480a9e19706633858d5b: Deleting consensus metadata
I20260923 15:13:54.027671 2600 catalog_manager.cc:4841] TS 6788dea5efa2480a9e19706633858d5b (127.21.204.67:38889): tablet f60080973a20415ca15998dc9d2a25e6 (table test-table [id=fae07ca2de78461daf754a464f5e0db2]) successfully deleted
I20260923 15:13:54.029206 3136 ts_tablet_manager.cc:1918] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:13:54.029337 3136 log.cc:1196] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/f60080973a20415ca15998dc9d2a25e6
I20260923 15:13:54.029644 3136 ts_tablet_manager.cc:1939] T f60080973a20415ca15998dc9d2a25e6 P 1cfedee610644b8a9babca97e4463054: Deleting consensus metadata
W20260923 15:13:54.031598 2969 connection.cc:535] server connection from 127.21.204.65:51529 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 15:13:54.032141 2599 catalog_manager.cc:4841] TS 1cfedee610644b8a9babca97e4463054 (127.21.204.66:38565): tablet f60080973a20415ca15998dc9d2a25e6 (table test-table [id=fae07ca2de78461daf754a464f5e0db2]) successfully deleted
W20260923 15:13:54.032533 2601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:43391 (error 108)
I20260923 15:13:54.033427 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2817
I20260923 15:13:54.033449 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:54.033301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:43391 (local address 127.0.0.1:60330)
0923 15:13:54.033353 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:54.033411 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 15:13:54.034612 2601 catalog_manager.cc:4584] TS aa4e774a602c47b1bcf7e871e0fa5dd8 (127.21.204.65:43391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f60080973a20415ca15998dc9d2a25e6: Network error: Client connection negotiation failed: client connection to 127.21.204.65:43391: connect: Connection refused (error 111)
I20260923 15:13:54.040086 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2950
I20260923 15:13:54.047209 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 2582
I20260923 15:13:54.055105 22321 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 15:13:54.399571 22321 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 15:13:54.399669 22321 mini_ranger.cc:66] Stopping Ranger...
I20260923 15:13:54.744135 22321 mini_ranger.cc:68] Stopped Ranger
2026-09-23 15:13:54.744 UTC [1649] LOG: received smart shutdown request
2026-09-23 15:13:54.748 UTC [1649] LOG: background worker "logical replication launcher" (PID 1658) exited with exit code 1
2026-09-23 15:13:54.748 UTC [1653] LOG: shutting down
2026-09-23 15:13:54.926 UTC [1649] LOG: database system is shut down
2026-09-23T15:13:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46623 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:13:55 dist-test-slave-t5lt krb5kdc[3149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:13:55 dist-test-slave-t5lt krb5kdc[3149](info): set up 2 sockets
Sep 23 15:13:55 dist-test-slave-t5lt krb5kdc[3149](info): commencing operation
krb5kdc: starting...
W20260923 15:13:56.091111 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:13:56.209286 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35223
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:35223,127.21.204.125:43693,127.21.204.124:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:56.294420 3162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:56.294669 3162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:56.294708 3162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:56.295854 3162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:13:56.295924 3162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:56.295966 3162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:56.296001 3162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:13:56.296027 3162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:13:56.297551 3162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:35223,127.21.204.125:43693,127.21.204.124:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:56.297847 3162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:56.298121 3162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:56.303350 3167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:56.303311 3170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:56.303385 3162 server_base.cc:1034] running on GCE node
W20260923 15:13:56.303349 3168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:56.303874 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:56.304199 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:56.305393 3162 hybrid_clock.cc:648] HybridClock initialized: now 1790176436305362 us; error 43 us; skew 500 ppm
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:56.312403 3162 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:13:56.312852 3162 webserver.cc:460] Webserver started at http://127.21.204.126:38473/ using document root <none> and password file <none>
I20260923 15:13:56.313095 3162 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:56.313169 3162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:56.313344 3162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:13:56.314494 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "dbcd4f352e024db4baf324b766dd0328"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "6e427fa2295a1badc4e7abcb5fa0b1d9"
server_key_iv: "1389f70de443b0c1fbd9d6c5ad9602db"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.314862 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "dbcd4f352e024db4baf324b766dd0328"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "6e427fa2295a1badc4e7abcb5fa0b1d9"
server_key_iv: "1389f70de443b0c1fbd9d6c5ad9602db"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.316274 3162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:13:56.317139 3177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:56.317392 3162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:56.317502 3162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "dbcd4f352e024db4baf324b766dd0328"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "6e427fa2295a1badc4e7abcb5fa0b1d9"
server_key_iv: "1389f70de443b0c1fbd9d6c5ad9602db"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.317590 3162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:56.326682 3162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:56.328529 3162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:56.328691 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:56.332825 3162 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:35223
I20260923 15:13:56.332880 3229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:35223 every 8 connection(s)
I20260923 15:13:56.333232 3162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:13:56.334203 3230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:56.334368 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3162
I20260923 15:13:56.334494 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:13:56.334700 22321 external_mini_cluster.cc:1428] Setting key 4468558803703187eecd81e1758a9bf3
I20260923 15:13:56.336360 3230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.125@KRBTEST.COM; defaulting to no policy
I20260923 15:13:56.348433 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.337218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:35223 (local address 127.0.0.1:43478)
0923 15:13:56.337385 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.337426 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.338075 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.338458 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.338461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.338836 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.339060 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.339066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.340563 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.340564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.341536 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.341539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.342235 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.342235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.342304 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.342860 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.342870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.344704 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.346877 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.346884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.346886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.347076 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.347374 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.347374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.347375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.347470 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.347885 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.347892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.347954 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.348046 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.348205 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260923 15:13:56.348448 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.337472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.21.204.126:35223)
0923 15:13:56.337613 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.337618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.337649 (+ 31us) server_negotiation.cc:407] Connection header received
0923 15:13:56.338269 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.338271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.338291 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.338360 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.339163 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.340469 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.341617 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.342052 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.342143 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.344809 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.344819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.344819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.344828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.346776 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.347169 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.347172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.347172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.347285 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.347561 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.347563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.347563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.347795 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.347878 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.348109 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.348294 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260923 15:13:56.349187 3230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:13:56.349532 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.349414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:58966)
0923 15:13:56.349464 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.349512 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 15:13:56.349743 3181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.125@KRBTEST.COM" created.
I20260923 15:13:56.350242 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.350125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:58972)
0923 15:13:56.350176 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.350209 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 15:13:56.350346 3230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:43693: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab.
Entry for principal kudu/127.21.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab.
I20260923 15:13:56.365010 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:43693
--webserver_interface=127.21.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab
--principal=kudu/127.21.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:35223,127.21.204.125:43693,127.21.204.124:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:13:56.399573 3230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } attempt: 1
I20260923 15:13:56.400152 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.399959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:58976)
0923 15:13:56.400046 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.400103 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 15:13:56.400648 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.400485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:58990)
0923 15:13:56.400575 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.400615 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 15:13:56.400919 3230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:43693: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
W20260923 15:13:56.446008 3237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:56.446214 3237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:56.446236 3237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:56.447322 3237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:13:56.447356 3237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:56.447371 3237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:56.447386 3237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:13:56.447397 3237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:13:56.448804 3237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:35223,127.21.204.125:43693,127.21.204.124:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab
--principal=kudu/127.21.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:56.449038 3237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:56.449249 3237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:56.453896 3242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:56.453953 3245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:56.453953 3243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:56.454221 3237 server_base.cc:1034] running on GCE node
I20260923 15:13:56.454442 3237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:56.454669 3237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:56.455829 3237 hybrid_clock.cc:648] HybridClock initialized: now 1790176436455811 us; error 40 us; skew 500 ppm
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:56.458331 3237 init.cc:377] Logged in from keytab as kudu/127.21.204.125@KRBTEST.COM (short username kudu)
I20260923 15:13:56.458818 3237 webserver.cc:460] Webserver started at http://127.21.204.125:34623/ using document root <none> and password file <none>
I20260923 15:13:56.459056 3237 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:56.459107 3237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:56.459288 3237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:13:56.460304 3237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data/instance:
uuid: "5f28586fe1a546eb9a76df52c5ace465"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "6a85c54dbbc9affb2a217c6310c74395"
server_key_iv: "078bd250b7352e51056b261934155f76"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.460667 3237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/wal/instance:
uuid: "5f28586fe1a546eb9a76df52c5ace465"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "6a85c54dbbc9affb2a217c6310c74395"
server_key_iv: "078bd250b7352e51056b261934155f76"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.462006 3237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:13:56.462795 3252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:56.462977 3237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:13:56.463048 3237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/wal
uuid: "5f28586fe1a546eb9a76df52c5ace465"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "6a85c54dbbc9affb2a217c6310c74395"
server_key_iv: "078bd250b7352e51056b261934155f76"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.463150 3237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:56.469089 3230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } attempt: 2
I20260923 15:13:56.469518 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.469393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:59002)
0923 15:13:56.469450 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.469487 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 15:13:56.469826 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.469748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:59008)
0923 15:13:56.469787 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.469810 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 15:13:56.469939 3230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:43693: Network error: Client connection negotiation failed: client connection to 127.21.204.125:43693: connect: Connection refused (error 111)
I20260923 15:13:56.474795 3237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:56.476682 3237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:56.476795 3237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:56.480882 3237 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.125:43693
I20260923 15:13:56.480935 3304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.125:43693 every 8 connection(s)
I20260923 15:13:56.481267 3237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
I20260923 15:13:56.482154 3305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:56.483665 3305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:56.489770 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3237
I20260923 15:13:56.489926 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-1/wal/instance
I20260923 15:13:56.490136 22321 external_mini_cluster.cc:1428] Setting key 40afef6791e385d1000b56493aed69bf
I20260923 15:13:56.492698 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.484372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.21.204.126:35223)
0923 15:13:56.484418 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.484420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.484653 (+ 233us) server_negotiation.cc:407] Connection header received
0923 15:13:56.485114 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.485116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.485130 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.485191 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.485802 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.486383 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.487350 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.487549 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.487603 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.489697 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.489699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.489699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.489706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.491177 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.491658 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.491659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.491660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.491729 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.491956 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.491957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.491957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.492102 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.492185 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.492473 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.492568 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:13:56.492749 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.484328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:35223 (local address 127.0.0.1:43480)
0923 15:13:56.484497 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.484538 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.485008 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.485263 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.485266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.485504 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.485702 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.485705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.486480 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.486481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.487272 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.487274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.487624 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.487625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.487726 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.488169 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.488178 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.489577 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.491261 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.491268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.491271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.491583 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.491800 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.491801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.491801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.491894 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.492196 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.492204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.492326 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.492431 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.492562 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260923 15:13:56.493376 3305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.124@KRBTEST.COM" created.
I20260923 15:13:56.501577 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.493648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:59022)
0923 15:13:56.493817 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.493827 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.493899 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.494116 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.494117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.494283 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.494342 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.494344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.495270 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.495271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.496136 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.496140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.496615 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.496616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.496698 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.497085 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.497090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.498483 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.500293 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.500294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.500294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.500510 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.500728 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.500730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.500730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.500828 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.501209 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.501210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.501253 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.501342 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.501460 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260923 15:13:56.501672 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.493690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.21.204.125:43693)
0923 15:13:56.493833 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.493837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.493919 (+ 82us) server_negotiation.cc:407] Connection header received
0923 15:13:56.493993 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.493996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.494019 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.494091 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.494500 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.495176 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.496241 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.496533 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.496588 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.498580 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.498588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.498588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.498595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.500187 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.500590 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.500592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.500592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.500644 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.500917 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.500918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.500918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.501107 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.501216 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.501391 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.501517 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260923 15:13:56.502287 3305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:13:56.502689 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.502556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41104)
0923 15:13:56.502614 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.502668 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 15:13:56.502833 3254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
I20260923 15:13:56.503273 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.503179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41112)
0923 15:13:56.503227 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.503250 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 15:13:56.503379 3305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:41885: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab.
Entry for principal kudu/127.21.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab.
I20260923 15:13:56.514814 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:41885
--webserver_interface=127.21.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab
--principal=kudu/127.21.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:35223,127.21.204.125:43693,127.21.204.124:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:13:56.552604 3305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } attempt: 1
I20260923 15:13:56.553134 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.552970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41126)
0923 15:13:56.553034 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.553102 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 15:13:56.553473 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.553371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41136)
0923 15:13:56.553416 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.553441 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 15:13:56.553576 3305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:41885: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
I20260923 15:13:56.561100 3230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } attempt: 3
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
I20260923 15:13:56.568619 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.561541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.21.204.125:43693)
0923 15:13:56.561612 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.561614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.561625 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:13:56.561709 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.561710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.561721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.561785 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.562174 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.562698 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.563399 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.563613 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.563721 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.565829 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.565830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.565830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.565837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.567497 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.567881 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.567881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.567882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.567931 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.568122 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.568122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.568123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.568222 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.568271 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.568482 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.568558 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 15:13:56.568622 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.561468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:59038)
0923 15:13:56.561542 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.561552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.561616 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.561793 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.561794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.562017 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.562080 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.562083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.562785 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.562786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.563313 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.563315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.563719 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.563720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.563807 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.564270 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.564283 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.565744 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.567586 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.567587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.567587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.567804 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.567996 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.567996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.567997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.568062 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.568285 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.568286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.568339 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.568412 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.568511 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 15:13:56.569361 3230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:13:56.569744 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.569620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41140)
0923 15:13:56.569671 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.569716 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 15:13:56.570140 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.570035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41144)
0923 15:13:56.570093 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.570120 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 15:13:56.570250 3230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:41885: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
W20260923 15:13:56.597635 3312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:56.597832 3312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:56.597855 3312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:56.599020 3312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:13:56.599067 3312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:56.599082 3312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:56.599094 3312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:13:56.599107 3312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:13:56.600572 3312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:35223,127.21.204.125:43693,127.21.204.124:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab
--principal=kudu/127.21.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:56.600816 3312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:56.601065 3312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:13:56.601418 3230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } attempt: 1
I20260923 15:13:56.601907 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.601724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41156)
0923 15:13:56.601801 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.601850 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 15:13:56.602357 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.602214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41158)
0923 15:13:56.602290 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.602325 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 15:13:56.602530 3230 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:41885: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
W20260923 15:13:56.606145 3318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:56.606204 3320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:56.606163 3317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:56.606343 3312 server_base.cc:1034] running on GCE node
I20260923 15:13:56.606631 3312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:56.606925 3312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:56.608107 3312 hybrid_clock.cc:648] HybridClock initialized: now 1790176436608083 us; error 40 us; skew 500 ppm
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:56.610769 3312 init.cc:377] Logged in from keytab as kudu/127.21.204.124@KRBTEST.COM (short username kudu)
I20260923 15:13:56.611244 3312 webserver.cc:460] Webserver started at http://127.21.204.124:42455/ using document root <none> and password file <none>
I20260923 15:13:56.611509 3312 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:56.611558 3312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:56.611712 3312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:13:56.612784 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data/instance:
uuid: "84212da2131e498f893eccd13db440d0"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "e166baf2db96a47f61c1a3dff657ff56"
server_key_iv: "fe01847703f2f14387a6307969fd0f69"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.613129 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/wal/instance:
uuid: "84212da2131e498f893eccd13db440d0"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "e166baf2db96a47f61c1a3dff657ff56"
server_key_iv: "fe01847703f2f14387a6307969fd0f69"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.614595 3312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:13:56.615458 3327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:56.615664 3312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:56.615764 3312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/wal
uuid: "84212da2131e498f893eccd13db440d0"
format_stamp: "Formatted at 2026-09-23 15:13:56 on dist-test-slave-t5lt"
server_key: "e166baf2db96a47f61c1a3dff657ff56"
server_key_iv: "fe01847703f2f14387a6307969fd0f69"
server_key_version: "encryptionkey@0"
I20260923 15:13:56.615825 3312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:56.621711 3305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } attempt: 2
I20260923 15:13:56.622184 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.622062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41168)
0923 15:13:56.622119 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.622155 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 15:13:56.622555 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.622466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41170)
0923 15:13:56.622511 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.622536 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 15:13:56.622660 3305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:41885: Network error: Client connection negotiation failed: client connection to 127.21.204.124:41885: connect: Connection refused (error 111)
I20260923 15:13:56.632846 3312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:56.635010 3312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:56.635164 3312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:56.639046 3312 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.124:41885
I20260923 15:13:56.639101 3379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.124:41885 every 8 connection(s)
I20260923 15:13:56.639423 3312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
I20260923 15:13:56.640450 3380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:56.642133 3380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:56.649238 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3312
I20260923 15:13:56.649348 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790176278407460-22321-0/minicluster-data/master-2/wal/instance
I20260923 15:13:56.649576 22321 external_mini_cluster.cc:1428] Setting key cb4c90d8f1bc8e554beb89f5dc7dd57c
I20260923 15:13:56.651398 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.642894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.21.204.126:35223)
0923 15:13:56.642970 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.642972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.643252 (+ 280us) server_negotiation.cc:407] Connection header received
0923 15:13:56.643707 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.643708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.643718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.643775 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.644389 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.644855 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.645784 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.646085 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.646144 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.648286 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.648287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.648288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.648294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.649807 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.650332 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.650333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.650334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.650419 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.650712 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.650713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.650713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.650830 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.650905 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.651227 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.651312 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 15:13:56.651575 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.642869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:35223 (local address 127.0.0.1:43494)
0923 15:13:56.643134 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.643170 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.643603 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.643857 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.643860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.644122 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.644304 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.644308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.644935 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.644936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.645703 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.645705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.646166 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.646167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.646270 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.646860 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.646868 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.648201 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.649917 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.649925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.649927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.650236 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.650503 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.650505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.650505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.650638 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.650908 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.650916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.651078 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.651188 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.651339 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260923 15:13:56.652124 3380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
I20260923 15:13:56.658529 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.650009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.21.204.124:41885)
0923 15:13:56.650207 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.650213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.650258 (+ 45us) server_negotiation.cc:407] Connection header received
0923 15:13:56.650329 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.650331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.650344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.650452 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.650997 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.651867 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.652863 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.653275 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.653348 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.655827 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.655835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.655835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.655842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.657343 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.657706 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.657708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.657709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.657769 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.657980 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.657981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.657981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.658130 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.658283 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.658338 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.658408 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
I20260923 15:13:56.659073 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.652372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:43693 (local address 127.0.0.1:59048)
0923 15:13:56.652489 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.652513 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.652563 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.652678 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.652678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.652822 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.652877 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.652878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.653636 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.653638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.654188 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.654190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.654539 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.654540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.654615 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.654986 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.654991 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.656213 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.657989 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.657990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.657991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.658181 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.658389 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.658390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.658391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.658446 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.658764 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.658765 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.658809 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.658884 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.658968 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 15:13:56.659139 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.652444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.21.204.125:43693)
0923 15:13:56.652487 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.652488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.652550 (+ 62us) server_negotiation.cc:407] Connection header received
0923 15:13:56.652620 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.652621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.652628 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.652667 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.652944 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.653559 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.654268 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.654463 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.654539 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.656340 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.656341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.656341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.656348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.657836 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.658266 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.658267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.658267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.658324 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.658547 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.658549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.658550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.658686 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.658763 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.658914 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.659027 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:13:56.659945 3380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:56.666695 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.660168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41174)
0923 15:13:56.660216 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.660226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.660261 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.660377 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.660377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.660477 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.660502 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.660503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.661163 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.661164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.661846 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.661849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.662323 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.662324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.662386 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.662676 (+ 290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.662681 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.663828 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.665623 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.665624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.665625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.665854 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.666104 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.666105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.666106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.666175 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.666381 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.666382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.666427 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.666499 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.666632 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 15:13:56.666872 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.660216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.21.204.124:41885)
0923 15:13:56.660248 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.660250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.660257 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:13:56.660304 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.660305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.660314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.660374 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.660563 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.661078 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.661940 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.662249 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.662299 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.663912 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.663913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.663914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.663921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.665526 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.665967 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.665968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.665969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.666029 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.666241 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.666242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.666242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.666321 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.666362 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.666531 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.666694 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":16}
I20260923 15:13:56.667586 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.659321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.21.204.126:35223)
0923 15:13:56.659359 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.659361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.659369 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:13:56.659388 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.659389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.659398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.659448 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.659971 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.660639 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.661429 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.661604 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.661652 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.664665 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.664666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.664666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.664672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.666431 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.666827 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.666828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.666828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.666882 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.667085 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.667086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.667087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.667221 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.667390 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.667472 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.667530 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:13:56.668450 3380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0: Bootstrap starting.
I20260923 15:13:56.669327 3380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:56.669624 3380 log.cc:824] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:56.670573 3380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0: No bootstrap required, opened a new log
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
I20260923 15:13:56.672008 3380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.672209 3380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:56.672258 3380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84212da2131e498f893eccd13db440d0, State: Initialized, Role: FOLLOWER
I20260923 15:13:56.672382 3380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.672691 3380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:13:56.672919 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.673014 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 15:13:56.674805 3397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:13:56.674891 3397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:13:56.675659 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.668209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.21.204.125:43693)
0923 15:13:56.668250 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.668252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.668261 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:13:56.668281 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.668282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.668290 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.668360 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.668804 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.669244 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.670090 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.670335 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.670408 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.672404 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.672405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.672406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.672413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.674017 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.674399 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.674400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.674400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.674468 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.674753 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.674755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.674755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.674908 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.675122 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.675527 (+ 405us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.675601 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:13:56.677068 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:13:56.677702 3230 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } attempt: 2
I20260923 15:13:56.675360 3312 master_runner.cc:186] Error getting master registration for 127.21.204.126:35223: OK, instance_id { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" instance_seqno: 1790176436331649 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 15:13:56.678277 3312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:13:56.686568 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.678005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41182)
0923 15:13:56.678111 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.678122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.678163 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.678432 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.678434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.678603 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.678650 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.678653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.679543 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.679544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.680339 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.680341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.680800 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.680801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.680857 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.681263 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.681268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.682767 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.685342 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.685343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.685343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.685602 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.685819 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.685820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.685820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.685902 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.686252 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.686253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.686296 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.686361 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.686444 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 15:13:56.687181 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.678251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.21.204.124:41885)
0923 15:13:56.678304 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.678306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.678313 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:13:56.678330 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.678331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.678340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.678387 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.678734 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.679452 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.680419 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.680702 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.680755 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.682862 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.682863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.682864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.682870 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.685260 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.685689 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.685690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.685691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.685748 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.685974 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.685975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.685975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.686161 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.686225 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.686996 (+ 771us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.687093 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:13:56.687436 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.678782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.21.204.126:35223)
0923 15:13:56.678819 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.678821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.679461 (+ 640us) server_negotiation.cc:407] Connection header received
0923 15:13:56.679568 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.679569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.679578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.679606 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.680159 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.680809 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.681850 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.682162 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.682205 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.684559 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.684560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.684560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.684567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.686190 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.686570 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.686570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.686571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.686648 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.686861 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.686862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.686862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.686972 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.687055 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.687268 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.687341 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:13:56.688251 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.679039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.21.204.124:41885)
0923 15:13:56.679334 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.679336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.679693 (+ 357us) server_negotiation.cc:407] Connection header received
0923 15:13:56.679714 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.679714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.679733 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.679788 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.680409 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.681171 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.682182 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.682508 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.682580 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.684527 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.684528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.684529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.684535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.686802 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.687206 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.687207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.687207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.687261 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.687452 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.687453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.687453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.687578 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.687661 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.688062 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.688136 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":78,"threads_started":1}
I20260923 15:13:56.688724 3230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: Bootstrap starting.
I20260923 15:13:56.689297 3230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:56.689512 3230 log.cc:824] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:56.690320 3230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: No bootstrap required, opened a new log
I20260923 15:13:56.692373 3230 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.692498 3230 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:56.692549 3230 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcd4f352e024db4baf324b766dd0328, State: Initialized, Role: FOLLOWER
I20260923 15:13:56.692737 3230 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.693024 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.693080 3230 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:13:56.693154 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:13:56.696048 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.678419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.21.204.125:43693)
0923 15:13:56.678451 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.678453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.678973 (+ 520us) server_negotiation.cc:407] Connection header received
0923 15:13:56.679096 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.679097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.679107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.679138 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.683672 (+ 4534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.684434 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.690913 (+ 6479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.691230 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.691305 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.692676 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.692677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.692678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.692686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.694447 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.694913 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.694914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.694914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.694974 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.695208 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.695209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.695210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.695343 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.695441 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.695882 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.695956 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 15:13:56.696252 3162 master_runner.cc:186] Error getting master registration for 127.21.204.125:43693: OK, instance_id { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" instance_seqno: 1790176436479763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 15:13:56.698377 3162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 15:13:56.694959 3419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:13:56.698633 3419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:13:56.713821 3305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } attempt: 3
Sep 23 15:13:56 dist-test-slave-t5lt krb5kdc[3149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176436, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:13:56.721676 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.714232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:41885 (local address 127.0.0.1:41196)
0923 15:13:56.714290 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:13:56.714304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:13:56.714359 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:13:56.714515 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:13:56.714516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:13:56.714687 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:13:56.714733 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.714735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.715478 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.715479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.715992 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:13:56.715994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.716839 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.716840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:13:56.716908 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.717377 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:13:56.717383 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:13:56.718679 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:13:56.720528 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.720529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.720529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.720700 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.721000 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:13:56.721001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:13:56.721001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:13:56.721064 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:13:56.721353 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:13:56.721354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:13:56.721398 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 15:13:56.721480 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 15:13:56.721577 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 15:13:56.721686 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:56.714269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.21.204.124:41885)
0923 15:13:56.714310 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:56.714313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:56.714342 (+ 29us) server_negotiation.cc:407] Connection header received
0923 15:13:56.714422 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:56.714423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:56.714443 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:56.714507 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:56.714813 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.715398 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.716092 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:56.716309 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:56.716389 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:56.718787 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:56.718789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:56.718789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:56.718796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:56.720431 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.720826 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.720827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.720828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.720930 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:56.721129 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:56.721130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:56.721130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:56.721257 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:56.721337 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:56.721527 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 15:13:56.721604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:13:56.723229 3305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: Bootstrap starting.
I20260923 15:13:56.723994 3305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:56.724220 3305 log.cc:824] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:56.725006 3305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: No bootstrap required, opened a new log
I20260923 15:13:56.726555 3305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.726686 3305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:56.726749 3305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f28586fe1a546eb9a76df52c5ace465, State: Initialized, Role: FOLLOWER
I20260923 15:13:56.726879 3305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.727175 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.727237 3305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:13:56.727313 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 15:13:56.727352 3258 tablet.cc:2369] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:13:56.729125 3433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:13:56.729251 3433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:13:56.731384 3237 master_runner.cc:263] No leader master found from master 5f28586fe1a546eb9a76df52c5ace465
I20260923 15:13:56.731442 3237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 15:13:56.745810 3386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:13:56.745949 3386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.746181 3386 leader_election.cc:288] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbcd4f352e024db4baf324b766dd0328 (127.21.204.126:35223), 5f28586fe1a546eb9a76df52c5ace465 (127.21.204.125:43693)
I20260923 15:13:56.746639 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84212da2131e498f893eccd13db440d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbcd4f352e024db4baf324b766dd0328" is_pre_election: true
I20260923 15:13:56.746780 3204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84212da2131e498f893eccd13db440d0 in term 0.
I20260923 15:13:56.747066 3330 leader_election.cc:302] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84212da2131e498f893eccd13db440d0, dbcd4f352e024db4baf324b766dd0328; no voters:
I20260923 15:13:56.747426 3279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84212da2131e498f893eccd13db440d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f28586fe1a546eb9a76df52c5ace465" is_pre_election: true
I20260923 15:13:56.747565 3279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84212da2131e498f893eccd13db440d0 in term 0.
I20260923 15:13:56.747720 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:13:56.747779 3386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:13:56.747802 3386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:56.748689 3386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.748801 3386 leader_election.cc:288] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [CANDIDATE]: Term 1 election: Requested vote from peers dbcd4f352e024db4baf324b766dd0328 (127.21.204.126:35223), 5f28586fe1a546eb9a76df52c5ace465 (127.21.204.125:43693)
I20260923 15:13:56.749243 3279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84212da2131e498f893eccd13db440d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f28586fe1a546eb9a76df52c5ace465"
I20260923 15:13:56.749334 3279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:56.749979 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84212da2131e498f893eccd13db440d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbcd4f352e024db4baf324b766dd0328"
I20260923 15:13:56.750070 3204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:56.750979 3204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84212da2131e498f893eccd13db440d0 in term 1.
I20260923 15:13:56.751189 3330 leader_election.cc:302] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84212da2131e498f893eccd13db440d0, dbcd4f352e024db4baf324b766dd0328; no voters:
I20260923 15:13:56.751659 3279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84212da2131e498f893eccd13db440d0 in term 1.
I20260923 15:13:56.751916 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:13:56.752089 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [term 1 LEADER]: Becoming Leader. State: Replica: 84212da2131e498f893eccd13db440d0, State: Running, Role: LEADER
I20260923 15:13:56.752185 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } }
I20260923 15:13:56.752559 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84212da2131e498f893eccd13db440d0. Latest consensus state: current_term: 1 leader_uuid: "84212da2131e498f893eccd13db440d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.752625 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:56.752920 3437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:13:56.754009 3437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:13:56.756067 3279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [term 1 FOLLOWER]: Refusing update from remote peer 84212da2131e498f893eccd13db440d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:13:56.756314 3204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [term 1 FOLLOWER]: Refusing update from remote peer 84212da2131e498f893eccd13db440d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:13:56.756783 3386 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:13:56.756919 3386 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:13:56.758811 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84212da2131e498f893eccd13db440d0. Latest consensus state: current_term: 1 leader_uuid: "84212da2131e498f893eccd13db440d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.758908 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:13:56.759667 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84212da2131e498f893eccd13db440d0. Latest consensus state: current_term: 1 leader_uuid: "84212da2131e498f893eccd13db440d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.759756 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:13:56.761507 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84212da2131e498f893eccd13db440d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.761585 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:56.761977 3408 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84212da2131e498f893eccd13db440d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.762238 3408 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:13:56.762488 3438 mvcc.cc:205] Tried to move back new op lower bound from 7332562684950188032 to 7332562684937494528. Current Snapshot: MvccSnapshot[applied={T|T < 7332562684950188032}]
I20260923 15:13:56.763180 3441 mvcc.cc:205] Tried to move back new op lower bound from 7332562684950188032 to 7332562684937494528. Current Snapshot: MvccSnapshot[applied={T|T < 7332562684950188032}]
I20260923 15:13:56.763649 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84212da2131e498f893eccd13db440d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.763721 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:56.764736 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84212da2131e498f893eccd13db440d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcd4f352e024db4baf324b766dd0328" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35223 } } peers { permanent_uuid: "5f28586fe1a546eb9a76df52c5ace465" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 43693 } } peers { permanent_uuid: "84212da2131e498f893eccd13db440d0" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 41885 } } }
I20260923 15:13:56.764815 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:13:56.765591 3439 mvcc.cc:205] Tried to move back new op lower bound from 7332562684950188032 to 7332562684937494528. Current Snapshot: MvccSnapshot[applied={T|T < 7332562684950188032}]
I20260923 15:13:56.766396 3437 catalog_manager.cc:1347] Generated new cluster ID: 3c5d6e289754464f91e76c11a3063182
I20260923 15:13:56.766441 3437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:13:56.790733 3437 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:13:56.791527 3437 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:13:56.799731 3437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84212da2131e498f893eccd13db440d0: Generated new TSK 0
I20260923 15:13:56.799983 3437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:13:57.699153 3419 catalog_manager.cc:1259] Loaded cluster ID: 3c5d6e289754464f91e76c11a3063182
I20260923 15:13:57.699263 3419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: loading cluster ID for follower catalog manager: success
I20260923 15:13:57.700090 3419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: acquiring CA information for follower catalog manager: success
I20260923 15:13:57.700428 3419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dbcd4f352e024db4baf324b766dd0328: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 15:13:57.729766 3433 catalog_manager.cc:1259] Loaded cluster ID: 3c5d6e289754464f91e76c11a3063182
I20260923 15:13:57.729841 3433 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: loading cluster ID for follower catalog manager: success
I20260923 15:13:57.730733 3433 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: acquiring CA information for follower catalog manager: success
I20260923 15:13:57.731051 3433 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f28586fe1a546eb9a76df52c5ace465: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 15:13:57.860925 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3162
I20260923 15:13:57.867233 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3237
I20260923 15:13:57.872562 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3312
2026-09-23T15:13:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2830 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:13:57 dist-test-slave-t5lt krb5kdc[3453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:13:57 dist-test-slave-t5lt krb5kdc[3453](info): set up 2 sockets
Sep 23 15:13:57 dist-test-slave-t5lt krb5kdc[3453](info): commencing operation
krb5kdc: starting...
W20260923 15:13:58.923216 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:13:59 dist-test-slave-t5lt krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:13:59.046104 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:39093
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:13:59.126919 3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:13:59.127158 3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:13:59.127194 3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:13:59.128341 3466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:13:59.128412 3466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:13:59.128436 3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:13:59.128463 3466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:13:59.128494 3466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:13:59.130002 3466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:39093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:13:59.130296 3466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:13:59.130529 3466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:13:59.135434 3472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:59.135423 3474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:13:59.135423 3471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:13:59.135457 3466 server_base.cc:1034] running on GCE node
I20260923 15:13:59.135908 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:13:59.136147 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:13:59.137292 3466 hybrid_clock.cc:648] HybridClock initialized: now 1790176439137270 us; error 37 us; skew 500 ppm
Sep 23 15:13:59 dist-test-slave-t5lt krb5kdc[3453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176439, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:13:59.139786 3466 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:13:59.140220 3466 webserver.cc:460] Webserver started at http://127.21.204.126:39413/ using document root <none> and password file <none>
I20260923 15:13:59.140468 3466 fs_manager.cc:359] Metadata directory not provided
I20260923 15:13:59.140522 3466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:13:59.140710 3466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:13:59.142012 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "155308dd057c41b5b8cdb1487f8676da"
format_stamp: "Formatted at 2026-09-23 15:13:59 on dist-test-slave-t5lt"
server_key: "7681907cdf7256b1f65602b908fb4875"
server_key_iv: "64cfc24731c6f2e4cfe70e8a98df60fc"
server_key_version: "encryptionkey@0"
I20260923 15:13:59.142403 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "155308dd057c41b5b8cdb1487f8676da"
format_stamp: "Formatted at 2026-09-23 15:13:59 on dist-test-slave-t5lt"
server_key: "7681907cdf7256b1f65602b908fb4875"
server_key_iv: "64cfc24731c6f2e4cfe70e8a98df60fc"
server_key_version: "encryptionkey@0"
I20260923 15:13:59.143741 3466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:59.144554 3481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:13:59.144807 3466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:13:59.144912 3466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "155308dd057c41b5b8cdb1487f8676da"
format_stamp: "Formatted at 2026-09-23 15:13:59 on dist-test-slave-t5lt"
server_key: "7681907cdf7256b1f65602b908fb4875"
server_key_iv: "64cfc24731c6f2e4cfe70e8a98df60fc"
server_key_version: "encryptionkey@0"
I20260923 15:13:59.144987 3466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:13:59.153609 3466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:13:59.155494 3466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:13:59.155650 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:13:59.159636 3466 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:39093
I20260923 15:13:59.159706 3533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:39093 every 8 connection(s)
I20260923 15:13:59.160084 3466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:13:59.160995 3534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:13:59.161008 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3466
I20260923 15:13:59.161123 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:13:59.161301 22321 external_mini_cluster.cc:1428] Setting key 5cabba56f5587c9bdc7c289322d1625f
I20260923 15:13:59.164181 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da: Bootstrap starting.
I20260923 15:13:59.164921 3534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da: Neither blocks nor log segments found. Creating new log.
I20260923 15:13:59.165205 3534 log.cc:824] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da: Log is configured to *not* fsync() on all Append() calls
I20260923 15:13:59.165859 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da: No bootstrap required, opened a new log
Sep 23 15:13:59 dist-test-slave-t5lt krb5kdc[3453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:13:59.168157 3534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155308dd057c41b5b8cdb1487f8676da" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 39093 } }
I20260923 15:13:59.168284 3534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:13:59.168306 3534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155308dd057c41b5b8cdb1487f8676da, State: Initialized, Role: FOLLOWER
I20260923 15:13:59.168452 3534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155308dd057c41b5b8cdb1487f8676da" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 39093 } }
I20260923 15:13:59.168515 3534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:13:59.168601 3534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:13:59.168653 3534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:13:59.169324 3534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155308dd057c41b5b8cdb1487f8676da" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 39093 } }
I20260923 15:13:59.169441 3534 leader_election.cc:302] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 155308dd057c41b5b8cdb1487f8676da; no voters:
I20260923 15:13:59.169624 3534 leader_election.cc:288] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:13:59.169660 3539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:13:59.169835 3539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [term 1 LEADER]: Becoming Leader. State: Replica: 155308dd057c41b5b8cdb1487f8676da, State: Running, Role: LEADER
I20260923 15:13:59.169917 3534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:13:59.171267 3539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155308dd057c41b5b8cdb1487f8676da" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 39093 } }
I20260923 15:13:59.171813 3540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 155308dd057c41b5b8cdb1487f8676da. Latest consensus state: current_term: 1 leader_uuid: "155308dd057c41b5b8cdb1487f8676da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155308dd057c41b5b8cdb1487f8676da" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 39093 } } }
I20260923 15:13:59.171859 3541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "155308dd057c41b5b8cdb1487f8676da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "155308dd057c41b5b8cdb1487f8676da" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 39093 } } }
I20260923 15:13:59.171949 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:13:59.161802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.21.204.126:39093)
0923 15:13:59.162007 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 15:13:59.162015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:13:59.162067 (+ 52us) server_negotiation.cc:407] Connection header received
0923 15:13:59.162493 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:13:59.162499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:13:59.162545 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:13:59.162653 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:13:59.163448 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:59.164340 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:59.165403 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:13:59.165787 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:13:59.165932 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:13:59.168473 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:13:59.168477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:13:59.168477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:13:59.168493 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:13:59.170370 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:59.170722 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:59.170725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:59.170726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:59.170820 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:13:59.171027 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:13:59.171027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:13:59.171028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:13:59.171255 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:13:59.171402 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 15:13:59.171513 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 15:13:59.171659 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260923 15:13:59.172000 3540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [sys.catalog]: This master's current role is: LEADER
I20260923 15:13:59.172012 3541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da [sys.catalog]: This master's current role is: LEADER
W20260923 15:13:59.172276 3553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:13:59.172346 3553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:13:59.172685 3555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:13:59.172832 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:13:59.172924 3555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:13:59.174476 3555 catalog_manager.cc:1347] Generated new cluster ID: 3f7cd496726b43a38b8877311b72df10
I20260923 15:13:59.174535 3555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:13:59.186072 3555 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:13:59.186555 3555 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:13:59.189833 3555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 155308dd057c41b5b8cdb1487f8676da: Generated new TSK 0
I20260923 15:13:59.190106 3555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:13:59.227265 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3466
2026-09-23T15:13:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1352 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:13:59 dist-test-slave-t5lt krb5kdc[3561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:13:59 dist-test-slave-t5lt krb5kdc[3561](info): set up 2 sockets
Sep 23 15:13:59 dist-test-slave-t5lt krb5kdc[3561](info): commencing operation
krb5kdc: starting...
W20260923 15:14:00.267349 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:14:00 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:14:00.388405 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41409
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:00.467816 3574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:00.468019 3574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:00.468041 3574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:00.469211 3574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:14:00.469254 3574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:00.469269 3574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:00.469281 3574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:14:00.469321 3574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:14:00.470837 3574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:00.471088 3574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:00.471335 3574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:00.476567 3580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:00.476580 3582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:00.476720 3574 server_base.cc:1034] running on GCE node
W20260923 15:14:00.476671 3579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:00.476976 3574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:00.477301 3574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:00.478503 3574 hybrid_clock.cc:648] HybridClock initialized: now 1790176440478490 us; error 41 us; skew 500 ppm
Sep 23 15:14:00 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176440, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:00.481274 3574 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:14:00.481794 3574 webserver.cc:460] Webserver started at http://127.21.204.126:34711/ using document root <none> and password file <none>
I20260923 15:14:00.482079 3574 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:00.482155 3574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:00.482307 3574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:00.483369 3574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "916928e1a565483a93532f4dafdc84b0"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "9a928352092cf53e01f6b2b3ede9dea0"
server_key_iv: "87c520aeb353612d84ea355f06f5b284"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.483726 3574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "916928e1a565483a93532f4dafdc84b0"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "9a928352092cf53e01f6b2b3ede9dea0"
server_key_iv: "87c520aeb353612d84ea355f06f5b284"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.485071 3574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:14:00.485795 3589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:00.485980 3574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:14:00.486088 3574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "916928e1a565483a93532f4dafdc84b0"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "9a928352092cf53e01f6b2b3ede9dea0"
server_key_iv: "87c520aeb353612d84ea355f06f5b284"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.486172 3574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:00.503540 3574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:00.505661 3574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:00.505831 3574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:00.510320 3574 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:41409
I20260923 15:14:00.510371 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:41409 every 8 connection(s)
I20260923 15:14:00.510737 3574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:14:00.511705 3642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:00.512624 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3574
I20260923 15:14:00.512745 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:14:00.512935 22321 external_mini_cluster.cc:1428] Setting key b0b8a9782306df142bdc9899c7c3f48a
I20260923 15:14:00.514560 3642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Bootstrap starting.
I20260923 15:14:00.515152 3642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:00.515443 3642 log.cc:824] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:00.516122 3642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: No bootstrap required, opened a new log
I20260923 15:14:00.517853 3642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:00.518102 3642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:00.518178 3642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916928e1a565483a93532f4dafdc84b0, State: Initialized, Role: FOLLOWER
I20260923 15:14:00.518337 3642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:00.518446 3642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:14:00.518492 3642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:14:00.518573 3642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:00.519085 3642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:00.519202 3642 leader_election.cc:302] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916928e1a565483a93532f4dafdc84b0; no voters:
I20260923 15:14:00.519374 3642 leader_election.cc:288] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:14:00.519421 3647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:14:00.519587 3647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 LEADER]: Becoming Leader. State: Replica: 916928e1a565483a93532f4dafdc84b0, State: Running, Role: LEADER
I20260923 15:14:00.519665 3642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:14:00.519737 3647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
Sep 23 15:14:00 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:00.520169 3647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916928e1a565483a93532f4dafdc84b0. Latest consensus state: current_term: 1 leader_uuid: "916928e1a565483a93532f4dafdc84b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } } }
I20260923 15:14:00.520300 3647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:00.520328 3649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "916928e1a565483a93532f4dafdc84b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } } }
I20260923 15:14:00.520406 3649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:00.520562 3652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:14:00.521625 3652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:14:00.523465 3652 catalog_manager.cc:1347] Generated new cluster ID: c80f993f08094b03b1e92fca59511c41
I20260923 15:14:00.523713 3652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:14:00.524693 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:00.513480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.21.204.126:41409)
0923 15:14:00.513692 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:00.513702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:00.514067 (+ 365us) server_negotiation.cc:407] Connection header received
0923 15:14:00.514582 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:00.514589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:00.514672 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:00.514823 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:00.515493 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.516492 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.517248 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.517659 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.517782 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:00.520935 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:00.520939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:00.520939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:00.520952 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:00.523063 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:00.523448 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:00.523452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:00.523452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:00.523583 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:00.523828 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:00.523829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:00.523829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:00.524064 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:00.524218 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:00.524323 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 15:14:00.524470 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260923 15:14:00.542598 3652 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:14:00.543332 3652 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:14:00.552181 3652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Generated new TSK 0
I20260923 15:14:00.552398 3652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:14:00.589242 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41409
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:00.675824 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:00.676026 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:00.676051 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:00.677129 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:00.677163 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:00.677189 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:14:00.678675 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:00.678864 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:00.679030 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:00.684170 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:00.684161 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:00.684162 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:00.684397 3668 server_base.cc:1034] running on GCE node
I20260923 15:14:00.684629 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:00.684841 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:00.686028 3668 hybrid_clock.cc:648] HybridClock initialized: now 1790176440686023 us; error 36 us; skew 500 ppm
Sep 23 15:14:00 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176440, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:00.688470 3668 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:14:00.688962 3668 webserver.cc:460] Webserver started at http://127.21.204.65:36119/ using document root <none> and password file <none>
I20260923 15:14:00.689210 3668 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:00.689281 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:00.689429 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:00.690542 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "daa7cdceafab4fd6904356d00bdd53d7"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "66f9dd56c562f8be7f553f8cc4501407"
server_key_iv: "95992d4af8d79b28f92ada8c5280446e"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.690899 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "daa7cdceafab4fd6904356d00bdd53d7"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "66f9dd56c562f8be7f553f8cc4501407"
server_key_iv: "95992d4af8d79b28f92ada8c5280446e"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.692214 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:14:00.692931 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:00.693117 3668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:14:00.693216 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "daa7cdceafab4fd6904356d00bdd53d7"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "66f9dd56c562f8be7f553f8cc4501407"
server_key_iv: "95992d4af8d79b28f92ada8c5280446e"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.693296 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:00.716774 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:00.719022 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:00.719200 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:00.719445 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:00.719847 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:00.719903 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:00.719946 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:00.719973 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:00.726366 3668 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:44879
I20260923 15:14:00.726413 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:44879 every 8 connection(s)
I20260923 15:14:00.726728 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
Sep 23 15:14:00 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176440, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:00.734892 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3668
I20260923 15:14:00.735000 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:14:00.735240 22321 external_mini_cluster.cc:1428] Setting key 4cd3f77cef48d294557f15a6ee7a3e2d
I20260923 15:14:00.736162 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:00.727447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:49047 (local address 127.21.204.126:41409)
0923 15:14:00.727539 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:00.727542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:00.727700 (+ 158us) server_negotiation.cc:407] Connection header received
0923 15:14:00.728210 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:00.728212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:00.728224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:00.728288 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:00.729219 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.729739 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.730797 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.731016 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.731079 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:00.733164 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:00.733165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:00.733165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:00.733172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:00.734652 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:00.735123 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:00.735124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:00.735125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:00.735177 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:00.735469 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:00.735470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:00.735470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:00.735605 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:00.735681 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:00.735977 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 15:14:00.736066 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 15:14:00.736404 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:00.727393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:49047)
0923 15:14:00.727576 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:00.727613 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:00.728099 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:00.728347 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:00.728350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:00.728919 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:00.729112 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:00.729116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.729847 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.729848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:00.730699 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:00.730701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.731097 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.731098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:00.731197 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:00.731651 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:00.731659 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:00.733008 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:00.734730 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:00.734738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:00.734740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:00.735041 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:00.735251 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:00.735252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:00.735252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:00.735391 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:00.735696 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:00.735703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:00.735842 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 15:14:00.735954 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 15:14:00.736130 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260923 15:14:00.736976 3797 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41409
I20260923 15:14:00.737087 3797 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:00.737301 3797 heartbeater.cc:511] Master 127.21.204.126:41409 requested a full tablet report, sending...
I20260923 15:14:00.737865 3606 ts_manager.cc:194] Registered new tserver with Master: daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65:44879)
I20260923 15:14:00.738531 3606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:49047
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:14:00.762506 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41409
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:00.842867 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:00.843067 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:00.843091 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:00.844233 3802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:00.844276 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:00.844301 3802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:14:00.845715 3802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:00.845978 3802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:00.846216 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:00.851226 3807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:00.851289 3802 server_base.cc:1034] running on GCE node
W20260923 15:14:00.851218 3808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:00.851218 3810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:00.851747 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:00.851982 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:00.853153 3802 hybrid_clock.cc:648] HybridClock initialized: now 1790176440853141 us; error 31 us; skew 500 ppm
Sep 23 15:14:00 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176440, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:00.855679 3802 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:14:00.856094 3802 webserver.cc:460] Webserver started at http://127.21.204.66:34705/ using document root <none> and password file <none>
I20260923 15:14:00.856338 3802 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:00.856407 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:00.856580 3802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:00.857590 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "bf92d8c74f4d4b4194de19b158da0316"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "3daed68ff2174b1d9a1e4ee347411582"
server_key_iv: "e7d14495a9a19c6bc03d59a7898501ba"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.857928 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "bf92d8c74f4d4b4194de19b158da0316"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "3daed68ff2174b1d9a1e4ee347411582"
server_key_iv: "e7d14495a9a19c6bc03d59a7898501ba"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.859285 3802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:00.860055 3817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:00.860477 3802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:00.860584 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "bf92d8c74f4d4b4194de19b158da0316"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "3daed68ff2174b1d9a1e4ee347411582"
server_key_iv: "e7d14495a9a19c6bc03d59a7898501ba"
server_key_version: "encryptionkey@0"
I20260923 15:14:00.860661 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:00.880067 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:00.882174 3802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:00.882328 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:00.882571 3802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:00.882886 3802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:00.882941 3802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:00.882975 3802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:00.883023 3802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:00.889106 3802 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:46227
I20260923 15:14:00.889225 3930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:46227 every 8 connection(s)
I20260923 15:14:00.889575 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:14:00 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176440, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:00.897225 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3802
I20260923 15:14:00.897578 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:14:00.897861 22321 external_mini_cluster.cc:1428] Setting key 1784fca5d83d6137b03464c96d6b3fa8
I20260923 15:14:00.900031 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:00.890414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:37153 (local address 127.21.204.126:41409)
0923 15:14:00.890481 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:00.890484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:00.890728 (+ 244us) server_negotiation.cc:407] Connection header received
0923 15:14:00.891245 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:00.891246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:00.891256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:00.891316 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:00.892562 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.893161 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.894691 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.894972 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.895057 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:00.897117 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:00.897118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:00.897119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:00.897126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:00.898475 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:00.898900 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:00.898901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:00.898902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:00.898964 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:00.899241 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:00.899242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:00.899242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:00.899358 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:00.899436 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:00.899724 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 15:14:00.899844 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:14:00.900203 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:00.890399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:37153)
0923 15:14:00.890606 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:00.890643 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:00.891138 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:00.891405 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:00.891409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:00.892190 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:00.892466 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:00.892472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.893278 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.893279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:00.894582 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:00.894585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:00.895065 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:00.895066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:00.895204 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:00.895617 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:00.895625 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:00.896999 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:00.898577 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:00.898585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:00.898588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:00.898805 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:00.899047 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:00.899048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:00.899049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:00.899164 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:00.899426 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:00.899434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:00.899579 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 15:14:00.899685 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 15:14:00.899942 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":117,"threads_started":1}
I20260923 15:14:00.900823 3931 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41409
I20260923 15:14:00.900905 3931 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:00.901093 3931 heartbeater.cc:511] Master 127.21.204.126:41409 requested a full tablet report, sending...
I20260923 15:14:00.901482 3606 ts_manager.cc:194] Registered new tserver with Master: bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
I20260923 15:14:00.901818 3606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:37153
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:14:00.925179 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41409
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:01.009393 3936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:01.009635 3936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:01.009675 3936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:01.010878 3936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:01.010946 3936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:01.010991 3936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:14:01.012460 3936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:01.012729 3936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:01.012964 3936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:01.017994 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.017994 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:01.018186 3936 server_base.cc:1034] running on GCE node
W20260923 15:14:01.018278 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:01.018461 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:01.018659 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:01.019795 3936 hybrid_clock.cc:648] HybridClock initialized: now 1790176441019782 us; error 28 us; skew 500 ppm
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:01.022444 3936 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:14:01.022888 3936 webserver.cc:460] Webserver started at http://127.21.204.67:41997/ using document root <none> and password file <none>
I20260923 15:14:01.023128 3936 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:01.023173 3936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:01.023259 3936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:01.024195 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "a4651b86ef91410498814075e11c3595"
format_stamp: "Formatted at 2026-09-23 15:14:01 on dist-test-slave-t5lt"
server_key: "6b6a74290e789a138fe39183eeb19b33"
server_key_iv: "c40ae4ac7ddc6f8f8037b23638f91d30"
server_key_version: "encryptionkey@0"
I20260923 15:14:01.024484 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "a4651b86ef91410498814075e11c3595"
format_stamp: "Formatted at 2026-09-23 15:14:01 on dist-test-slave-t5lt"
server_key: "6b6a74290e789a138fe39183eeb19b33"
server_key_iv: "c40ae4ac7ddc6f8f8037b23638f91d30"
server_key_version: "encryptionkey@0"
I20260923 15:14:01.025870 3936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:14:01.026634 3951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.026814 3936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:14:01.026880 3936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "a4651b86ef91410498814075e11c3595"
format_stamp: "Formatted at 2026-09-23 15:14:01 on dist-test-slave-t5lt"
server_key: "6b6a74290e789a138fe39183eeb19b33"
server_key_iv: "c40ae4ac7ddc6f8f8037b23638f91d30"
server_key_version: "encryptionkey@0"
I20260923 15:14:01.026966 3936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:01.041041 3936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:01.043401 3936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:01.043551 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:01.043787 3936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:01.044106 3936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:01.044142 3936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.044163 3936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:01.044176 3936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.049988 3936 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:36281
I20260923 15:14:01.050046 4064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:36281 every 8 connection(s)
I20260923 15:14:01.050381 3936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:14:01.051688 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 3936
I20260923 15:14:01.051802 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:14:01.051998 22321 external_mini_cluster.cc:1428] Setting key 41405e032452b039a5c9bba9c49bb119
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:01.059394 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.051165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:38827 (local address 127.21.204.126:41409)
0923 15:14:01.051252 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.051255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.051429 (+ 174us) server_negotiation.cc:407] Connection header received
0923 15:14:01.052065 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.052067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.052082 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.052175 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:01.053304 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.053803 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.054847 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.055032 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.055084 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.057041 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:01.057042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:01.057042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:01.057048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:01.058305 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.058654 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.058655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.058655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.058704 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.058903 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.058903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.058903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.058997 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:01.059045 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.059265 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.059331 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 15:14:01.059594 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.051133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:38827)
0923 15:14:01.051313 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.051348 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.051946 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.052314 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.052319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.053020 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:01.053214 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.053218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.053939 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.053940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.054772 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.054775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.055104 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.055104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.055206 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.055627 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:01.055635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:01.056955 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:01.058373 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.058379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.058381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.058589 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.058759 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.058760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.058760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.058850 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.059048 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:01.059052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:01.059160 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.059238 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.059350 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260923 15:14:01.060072 4065 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41409
I20260923 15:14:01.060168 4065 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:01.060354 4065 heartbeater.cc:511] Master 127.21.204.126:41409 requested a full tablet report, sending...
I20260923 15:14:01.060755 3606 ts_manager.cc:194] Registered new tserver with Master: a4651b86ef91410498814075e11c3595 (127.21.204.67:36281)
I20260923 15:14:01.061142 3606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:38827
I20260923 15:14:01.063778 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:14:01.070652 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.065092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.21.204.126:41409)
0923 15:14:01.065157 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.065159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.065218 (+ 59us) server_negotiation.cc:407] Connection header received
0923 15:14:01.065253 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.065253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.065261 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.065304 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:01.065788 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.066319 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.066980 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.067158 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.067206 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.068291 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:01.068292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:01.068292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:01.068297 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:01.069489 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.069831 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.069832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.069832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.069943 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.070146 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.070147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.070147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.070284 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:01.070361 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.070509 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.070565 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 15:14:01.072286 3606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:14:01.073829 3606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:14:01.080454 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.076322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44879 (local address 127.0.0.1:37254)
0923 15:14:01.076461 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.076502 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.076582 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.077038 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.077041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.077043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:01.077145 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.077146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.078516 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.078517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.079384 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.079387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.080175 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.080176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.080253 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.080265 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.080334 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.080379 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":72,"thread_start_us":65,"threads_started":1}
I20260923 15:14:01.080510 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.076451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.21.204.65:44879)
0923 15:14:01.076859 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.076865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.076875 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:14:01.076917 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.076920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.076939 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.077016 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:01.077241 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.078397 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.079479 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.080106 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.080177 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.080231 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.080365 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.080427 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":93,"threads_started":1}
I20260923 15:14:01.080996 4080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.076705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46227 (local address 127.0.0.1:56562)
0923 15:14:01.076912 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.076920 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.076953 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.077441 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.077442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.077443 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:01.077495 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.077496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.078446 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.078447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.079738 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.079742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.080757 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.080758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.080814 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.080817 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.080889 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.080931 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":59,"threads_started":1}
I20260923 15:14:01.081296 3731 tablet_service.cc:1467] Processing CreateTablet for tablet fc9d300292684bc2af59194fce75d880 (DEFAULT_TABLE table=test-table [id=98bd3da96c544131a122138977f40d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:01.081491 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.076533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:36281 (local address 127.0.0.1:34432)
0923 15:14:01.076778 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.076788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.076840 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.077054 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.077054 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.077056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:01.077117 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.077119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.078547 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.078548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.079752 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.079754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.081259 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.081260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.081324 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.081328 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.081385 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.081426 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":79,"thread_start_us":73,"threads_started":1}
I20260923 15:14:01.081534 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.076614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.21.204.67:36281)
0923 15:14:01.076878 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.076883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.076894 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:14:01.076932 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.076934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.076949 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.077201 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:01.077264 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.078397 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.079863 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.081009 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.081257 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.081318 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.081408 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.081446 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":80,"threads_started":1}
I20260923 15:14:01.081656 3731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9d300292684bc2af59194fce75d880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:01.082226 3999 tablet_service.cc:1467] Processing CreateTablet for tablet fc9d300292684bc2af59194fce75d880 (DEFAULT_TABLE table=test-table [id=98bd3da96c544131a122138977f40d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:01.082580 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9d300292684bc2af59194fce75d880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:01.084398 4082 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap starting.
I20260923 15:14:01.084604 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.076827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.21.204.66:46227)
0923 15:14:01.077202 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.077206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.077216 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:14:01.077254 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.077256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.077274 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.077350 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:01.077570 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.078345 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.079835 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.080588 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.080661 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.080722 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.084440 (+ 3718us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.084515 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":97,"threads_started":1}
I20260923 15:14:01.085180 4082 tablet_bootstrap.cc:654] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:01.085286 3865 tablet_service.cc:1467] Processing CreateTablet for tablet fc9d300292684bc2af59194fce75d880 (DEFAULT_TABLE table=test-table [id=98bd3da96c544131a122138977f40d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:01.085662 3865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9d300292684bc2af59194fce75d880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:01.085790 4082 log.cc:824] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:01.086171 4083 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Bootstrap starting.
I20260923 15:14:01.086553 4082 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: No bootstrap required, opened a new log
I20260923 15:14:01.086637 4082 ts_tablet_manager.cc:1397] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:14:01.086797 4083 tablet_bootstrap.cc:654] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:01.087357 4083 log.cc:824] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:01.088126 4083 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: No bootstrap required, opened a new log
I20260923 15:14:01.088198 4083 ts_tablet_manager.cc:1397] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:14:01.088366 4082 raft_consensus.cc:348] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.088572 4082 raft_consensus.cc:374] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:01.088622 4082 raft_consensus.cc:729] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa7cdceafab4fd6904356d00bdd53d7, State: Initialized, Role: FOLLOWER
I20260923 15:14:01.088745 4082 consensus_queue.cc:260] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.089201 4082 ts_tablet_manager.cc:1428] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:14:01.089212 4087 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Bootstrap starting.
I20260923 15:14:01.089371 3797 heartbeater.cc:503] Master 127.21.204.126:41409 was elected leader, sending a full tablet report...
I20260923 15:14:01.090000 4087 tablet_bootstrap.cc:654] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:01.090513 4087 log.cc:824] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:01.090546 4083 raft_consensus.cc:348] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.090704 4083 raft_consensus.cc:374] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:01.090770 4083 raft_consensus.cc:729] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4651b86ef91410498814075e11c3595, State: Initialized, Role: FOLLOWER
I20260923 15:14:01.090880 4083 consensus_queue.cc:260] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.091142 4065 heartbeater.cc:503] Master 127.21.204.126:41409 was elected leader, sending a full tablet report...
I20260923 15:14:01.091151 4083 ts_tablet_manager.cc:1428] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:14:01.091692 4087 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: No bootstrap required, opened a new log
I20260923 15:14:01.091763 4087 ts_tablet_manager.cc:1397] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 15:14:01.093274 4087 raft_consensus.cc:348] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.093411 4087 raft_consensus.cc:374] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:01.093456 4087 raft_consensus.cc:729] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf92d8c74f4d4b4194de19b158da0316, State: Initialized, Role: FOLLOWER
I20260923 15:14:01.093572 4087 consensus_queue.cc:260] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.093815 4087 ts_tablet_manager.cc:1428] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 15:14:01.093868 3931 heartbeater.cc:503] Master 127.21.204.126:41409 was elected leader, sending a full tablet report...
W20260923 15:14:01.140278 3932 tablet.cc:2369] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:14:01.227460 3798 tablet.cc:2369] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:14:01.300984 4066 tablet.cc:2369] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:14:01.401180 4089 raft_consensus.cc:482] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:01.401319 4089 raft_consensus.cc:504] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.401772 4089 leader_election.cc:288] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65:44879), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
I20260923 15:14:01.405681 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.402103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:51359 (local address 127.21.204.65:44879)
0923 15:14:01.402183 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.402186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.402215 (+ 29us) server_negotiation.cc:407] Connection header received
0923 15:14:01.402314 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.402316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.402330 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.402401 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:01.402562 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.403322 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.404757 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.405309 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.405391 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.405400 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.405570 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.405607 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 15:14:01.405681 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.402292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:46885 (local address 127.21.204.66:46227)
0923 15:14:01.402336 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.402338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.402483 (+ 145us) server_negotiation.cc:407] Connection header received
0923 15:14:01.402538 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.402539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.402553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.402604 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:01.402741 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.403415 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.404756 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.405318 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.405391 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.405400 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.405570 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.405607 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:14:01.405682 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.402067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44879 (local address 127.21.204.67:51359)
0923 15:14:01.402128 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.402158 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.402235 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.402409 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.402410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.402412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:01.402488 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.402490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.403402 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.403403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.404647 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.404650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.405400 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.405401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.405488 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.405490 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.405548 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.405596 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":44}
I20260923 15:14:01.405689 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.402228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46227 (local address 127.21.204.67:46885)
0923 15:14:01.402385 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.402395 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.402447 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.402622 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.402623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.402624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:01.402673 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.402674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.403478 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.403479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.404647 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.404650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.405400 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.405401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.405485 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.405489 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.405548 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.405596 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":11,"thread_start_us":105,"threads_started":1}
I20260923 15:14:01.406096 3885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9d300292684bc2af59194fce75d880" candidate_uuid: "a4651b86ef91410498814075e11c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf92d8c74f4d4b4194de19b158da0316" is_pre_election: true
I20260923 15:14:01.406277 3885 raft_consensus.cc:2454] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4651b86ef91410498814075e11c3595 in term 0.
I20260923 15:14:01.406243 3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9d300292684bc2af59194fce75d880" candidate_uuid: "a4651b86ef91410498814075e11c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa7cdceafab4fd6904356d00bdd53d7" is_pre_election: true
I20260923 15:14:01.406394 3751 raft_consensus.cc:2454] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4651b86ef91410498814075e11c3595 in term 0.
I20260923 15:14:01.406523 3954 leader_election.cc:302] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4651b86ef91410498814075e11c3595, bf92d8c74f4d4b4194de19b158da0316; no voters:
I20260923 15:14:01.406652 4089 raft_consensus.cc:2781] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:14:01.406713 4089 raft_consensus.cc:482] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:14:01.406738 4089 raft_consensus.cc:3037] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:01.407471 4089 raft_consensus.cc:504] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.407606 4089 leader_election.cc:288] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [CANDIDATE]: Term 1 election: Requested vote from peers daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65:44879), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
I20260923 15:14:01.407946 3885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9d300292684bc2af59194fce75d880" candidate_uuid: "a4651b86ef91410498814075e11c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf92d8c74f4d4b4194de19b158da0316"
I20260923 15:14:01.407951 3751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9d300292684bc2af59194fce75d880" candidate_uuid: "a4651b86ef91410498814075e11c3595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa7cdceafab4fd6904356d00bdd53d7"
I20260923 15:14:01.408051 3885 raft_consensus.cc:3037] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:01.408056 3751 raft_consensus.cc:3037] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:01.408780 3885 raft_consensus.cc:2454] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4651b86ef91410498814075e11c3595 in term 1.
I20260923 15:14:01.408782 3751 raft_consensus.cc:2454] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4651b86ef91410498814075e11c3595 in term 1.
I20260923 15:14:01.409003 3954 leader_election.cc:302] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4651b86ef91410498814075e11c3595, bf92d8c74f4d4b4194de19b158da0316; no voters:
I20260923 15:14:01.409132 4089 raft_consensus.cc:2781] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:14:01.409312 4089 raft_consensus.cc:686] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 1 LEADER]: Becoming Leader. State: Replica: a4651b86ef91410498814075e11c3595, State: Running, Role: LEADER
I20260923 15:14:01.409438 4089 consensus_queue.cc:237] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.410334 3606 catalog_manager.cc:5495] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4651b86ef91410498814075e11c3595 (127.21.204.67). New cstate: current_term: 1 leader_uuid: "a4651b86ef91410498814075e11c3595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } health_report { overall_health: UNKNOWN } } }
I20260923 15:14:01.434312 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.431729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.21.204.67:36281)
0923 15:14:01.431797 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.431799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.431808 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:14:01.431905 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.431911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.431950 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.432071 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:14:01.432218 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.432696 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.433448 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.433653 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.433726 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.433921 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:14:01.434044 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:14:01.434148 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.434173 (+ 25us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.434193 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:14:01.436288 3751 raft_consensus.cc:1261] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Refusing update from remote peer a4651b86ef91410498814075e11c3595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:01.436290 3885 raft_consensus.cc:1261] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Refusing update from remote peer a4651b86ef91410498814075e11c3595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:01.436733 4092 consensus_queue.cc:1035] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:01.436923 4089 consensus_queue.cc:1035] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:01.439590 4096 mvcc.cc:205] Tried to move back new op lower bound from 7332562704119898112 to 7332562704013639680. Current Snapshot: MvccSnapshot[applied={T|T < 7332562704119898112}]
I20260923 15:14:01.442158 3606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:01.442251 3606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:01.442979 3606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Sending DeleteTablet for 3 replicas of tablet fc9d300292684bc2af59194fce75d880
I20260923 15:14:01.443289 3731 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9d300292684bc2af59194fce75d880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:01 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:37254
I20260923 15:14:01.443358 3999 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9d300292684bc2af59194fce75d880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:01 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260923 15:14:01.443420 3865 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9d300292684bc2af59194fce75d880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:01 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:56562
I20260923 15:14:01.443642 4107 tablet_replica.cc:330] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: stopping tablet replica
I20260923 15:14:01.443645 4106 tablet_replica.cc:330] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: stopping tablet replica
I20260923 15:14:01.443655 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3668
I20260923 15:14:01.443747 4106 raft_consensus.cc:2229] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:14:01.443893 4106 raft_consensus.cc:2258] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:01.444231 4107 raft_consensus.cc:2229] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:01.444356 4107 raft_consensus.cc:2258] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:01.444413 4106 ts_tablet_manager.cc:1905] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:01.444775 4107 ts_tablet_manager.cc:1905] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:01.446373 4107 ts_tablet_manager.cc:1918] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:01.446403 4106 ts_tablet_manager.cc:1918] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:01.446473 4106 log.cc:1196] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/fc9d300292684bc2af59194fce75d880
I20260923 15:14:01.446492 4107 log.cc:1196] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/fc9d300292684bc2af59194fce75d880
I20260923 15:14:01.446772 4107 ts_tablet_manager.cc:1939] T fc9d300292684bc2af59194fce75d880 P bf92d8c74f4d4b4194de19b158da0316: Deleting consensus metadata
I20260923 15:14:01.446802 4106 ts_tablet_manager.cc:1939] T fc9d300292684bc2af59194fce75d880 P a4651b86ef91410498814075e11c3595: Deleting consensus metadata
I20260923 15:14:01.447201 3592 catalog_manager.cc:4841] TS a4651b86ef91410498814075e11c3595 (127.21.204.67:36281): tablet fc9d300292684bc2af59194fce75d880 (table test-table [id=98bd3da96c544131a122138977f40d16]) successfully deleted
I20260923 15:14:01.447301 3592 catalog_manager.cc:4841] TS bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227): tablet fc9d300292684bc2af59194fce75d880 (table test-table [id=98bd3da96c544131a122138977f40d16]) successfully deleted
W20260923 15:14:01.449726 3592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:44879 (error 108)
I20260923 15:14:01.450331 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3802
I20260923 15:14:01.450392 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.450220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44879 (local address 127.0.0.1:37260)
0923 15:14:01.450288 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.450344 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 15:14:01.451126 3592 catalog_manager.cc:4584] TS daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65:44879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc9d300292684bc2af59194fce75d880: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44879: connect: Connection refused (error 111)
I20260923 15:14:01.455685 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3936
I20260923 15:14:01.460404 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 3574
I20260923 15:14:01.465821 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41409
--webserver_interface=127.21.204.126
--webserver_port=34711
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:01.545920 4109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:01.546115 4109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:01.546139 4109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:01.547287 4109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:14:01.547329 4109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:01.547344 4109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:01.547356 4109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:14:01.547369 4109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:14:01.548884 4109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=34711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:01.549130 4109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:01.549373 4109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:01.558141 4118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.558123 4116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.558152 4115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:01.558176 4109 server_base.cc:1034] running on GCE node
I20260923 15:14:01.558632 4109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:01.558884 4109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:01.560037 4109 hybrid_clock.cc:648] HybridClock initialized: now 1790176441560027 us; error 40 us; skew 500 ppm
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:01.565526 4109 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:14:01.565971 4109 webserver.cc:460] Webserver started at http://127.21.204.126:34711/ using document root <none> and password file <none>
I20260923 15:14:01.566177 4109 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:01.566227 4109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:01.567585 4109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:01.568323 4125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.568527 4109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:01.568641 4109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "916928e1a565483a93532f4dafdc84b0"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "9a928352092cf53e01f6b2b3ede9dea0"
server_key_iv: "87c520aeb353612d84ea355f06f5b284"
server_key_version: "encryptionkey@0"
I20260923 15:14:01.569046 4109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:01.584095 4109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:01.586058 4109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:01.586212 4109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:01.590248 4109 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:41409
I20260923 15:14:01.590299 4177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:41409 every 8 connection(s)
I20260923 15:14:01.590710 4109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:14:01.592239 4178 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:14:01.592921 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Bootstrap starting.
I20260923 15:14:01.600747 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 4109
I20260923 15:14:01.601049 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:44879
--local_ip_for_outbound_sockets=127.21.204.65
--tserver_master_addrs=127.21.204.126:41409
--webserver_port=36119
--webserver_interface=127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:14:01.601390 4178 log.cc:824] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:01.603282 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:14:01.603590 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Bootstrap complete.
I20260923 15:14:01.605829 4178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:01.606168 4178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916928e1a565483a93532f4dafdc84b0, State: Initialized, Role: FOLLOWER
I20260923 15:14:01.606352 4178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:01.606432 4178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:14:01.606519 4178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:14:01.606572 4178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 1 FOLLOWER]: Advancing to term 2
I20260923 15:14:01.607532 4178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:01.607653 4178 leader_election.cc:302] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916928e1a565483a93532f4dafdc84b0; no voters:
I20260923 15:14:01.607857 4178 leader_election.cc:288] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 15:14:01.607896 4182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 15:14:01.608011 4182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 LEADER]: Becoming Leader. State: Replica: 916928e1a565483a93532f4dafdc84b0, State: Running, Role: LEADER
I20260923 15:14:01.608127 4178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:14:01.608160 4182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:01.608430 4182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916928e1a565483a93532f4dafdc84b0. Latest consensus state: current_term: 2 leader_uuid: "916928e1a565483a93532f4dafdc84b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } } }
I20260923 15:14:01.608507 4182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:01.608435 4183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "916928e1a565483a93532f4dafdc84b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } } }
I20260923 15:14:01.608702 4183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:01.609861 4198 catalog_manager.cc:1259] Loaded cluster ID: c80f993f08094b03b1e92fca59511c41
I20260923 15:14:01.609985 4198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: loading cluster ID for follower catalog manager: success
W20260923 15:14:01.610177 4198 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260923 15:14:01.610232 4198 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260923 15:14:01.610275 4187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:14:01.610584 4187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:14:01.610682 4187 catalog_manager.cc:1259] Loaded cluster ID: c80f993f08094b03b1e92fca59511c41
I20260923 15:14:01.610736 4187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:14:01.610905 4187 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 15:14:01.613574 4187 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 15:14:01.615496 4187 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:14:01.615696 4187 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Loaded TSK: 0
I20260923 15:14:01.615911 4187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 15:14:01.697721 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:01.697932 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:01.698004 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:01.699141 4181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:01.699179 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:01.699205 4181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:14:01.700635 4181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:44879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=36119
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:01.700872 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:01.701098 4181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:01.706926 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.706956 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.706923 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:01.707149 4181 server_base.cc:1034] running on GCE node
I20260923 15:14:01.707398 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:01.707669 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:01.708845 4181 hybrid_clock.cc:648] HybridClock initialized: now 1790176441708827 us; error 32 us; skew 500 ppm
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:01.711360 4181 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:14:01.711767 4181 webserver.cc:460] Webserver started at http://127.21.204.65:36119/ using document root <none> and password file <none>
I20260923 15:14:01.711999 4181 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:01.712081 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:01.713521 4181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:01.714329 4215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.714505 4181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:14:01.714619 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "daa7cdceafab4fd6904356d00bdd53d7"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "66f9dd56c562f8be7f553f8cc4501407"
server_key_iv: "95992d4af8d79b28f92ada8c5280446e"
server_key_version: "encryptionkey@0"
I20260923 15:14:01.714949 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:01.753183 4181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:01.755630 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:01.755803 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:01.756060 4181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:01.756536 4222 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 15:14:01.757977 4181 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 15:14:01.758034 4181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 15:14:01.758081 4181 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 15:14:01.758724 4181 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 15:14:01.758780 4181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 15:14:01.758860 4222 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap starting.
I20260923 15:14:01.765723 4181 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:44879
I20260923 15:14:01.765820 4329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:44879 every 8 connection(s)
I20260923 15:14:01.766139 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:14:01.766772 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 4181
I20260923 15:14:01.767088 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:46227
--local_ip_for_outbound_sockets=127.21.204.66
--tserver_master_addrs=127.21.204.126:41409
--webserver_port=34705
--webserver_interface=127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:14:01.769660 4222 log.cc:824] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:01.772148 4222 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:14:01.772462 4222 tablet_bootstrap.cc:492] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap complete.
I20260923 15:14:01.773499 4222 ts_tablet_manager.cc:1397] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260923 15:14:01.774641 4222 raft_consensus.cc:348] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.774952 4222 raft_consensus.cc:729] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa7cdceafab4fd6904356d00bdd53d7, State: Initialized, Role: FOLLOWER
I20260923 15:14:01.775113 4222 consensus_queue.cc:260] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } } peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } }
I20260923 15:14:01.775477 4222 ts_tablet_manager.cc:1428] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:01.781095 4333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.767357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:32795 (local address 127.21.204.126:41409)
0923 15:14:01.770040 (+ 2683us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.770048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.770079 (+ 31us) server_negotiation.cc:407] Connection header received
0923 15:14:01.770185 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.770190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.770232 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.770392 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:01.771288 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.772796 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.774483 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.775069 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.775216 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.777846 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:01.777850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:01.777851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:01.777861 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:01.779477 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.779943 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.779946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.779946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.780029 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.780287 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.780288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.780288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.780484 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:01.780545 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.780853 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.780957 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2633,"thread_start_us":80,"threads_started":1}
I20260923 15:14:01.781165 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.766997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:32795)
0923 15:14:01.767466 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.767519 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.768228 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.770443 (+ 2215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.770448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.770853 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:01.771139 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.771147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.772915 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.772916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.774369 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.774373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.775198 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.775199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.775436 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.775985 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:01.775996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:01.777740 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:01.779562 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.779570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.779572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.779859 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.780095 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.780096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.780096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.780224 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.780546 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:01.780552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:01.780697 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.780805 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.780963 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":110,"threads_started":1}
I20260923 15:14:01.781855 4330 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41409
I20260923 15:14:01.782013 4330 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:01.782217 4330 heartbeater.cc:511] Master 127.21.204.126:41409 requested a full tablet report, sending...
I20260923 15:14:01.782943 4142 ts_manager.cc:194] Registered new tserver with Master: daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65:44879)
I20260923 15:14:01.784142 4142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:32795
I20260923 15:14:01.784997 4330 heartbeater.cc:503] Master 127.21.204.126:41409 was elected leader, sending a full tablet report...
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:14:01.791815 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.783538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44879 (local address 127.0.0.1:37264)
0923 15:14:01.783829 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.783855 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.783944 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.784128 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.784134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.784497 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:01.784575 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.784579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.785963 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.785964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.786523 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.786526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.787130 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.787131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.787198 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.787643 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:01.787647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:01.788859 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:01.790719 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.790727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.790728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.790916 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.791135 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.791136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.791136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.791200 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.791501 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:01.791507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:01.791546 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.791607 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.791746 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":73,"threads_started":1}
I20260923 15:14:01.791851 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.783606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.21.204.65:44879)
0923 15:14:01.783765 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.783769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.783925 (+ 156us) server_negotiation.cc:407] Connection header received
0923 15:14:01.784019 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.784021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.784036 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.784125 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:01.784660 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.785841 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.786604 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.787054 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.787123 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.788939 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:01.788947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:01.788947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:01.788954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:01.790644 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.790995 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.790998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.790998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.791068 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.791263 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.791264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.791264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.791434 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:01.791519 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.791661 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.791756 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260923 15:14:01.792469 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9d300292684bc2af59194fce75d880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:01 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:37264
I20260923 15:14:01.792979 4339 tablet_replica.cc:330] stopping tablet replica
I20260923 15:14:01.793083 4339 raft_consensus.cc:2229] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:01.793151 4339 raft_consensus.cc:2258] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:01.793431 4339 ts_tablet_manager.cc:1905] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:01.795382 4339 ts_tablet_manager.cc:1918] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:01.795444 4339 log.cc:1196] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/fc9d300292684bc2af59194fce75d880
I20260923 15:14:01.795661 4339 ts_tablet_manager.cc:1939] T fc9d300292684bc2af59194fce75d880 P daa7cdceafab4fd6904356d00bdd53d7: Deleting consensus metadata
I20260923 15:14:01.796136 4128 catalog_manager.cc:4841] TS daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65:44879): tablet fc9d300292684bc2af59194fce75d880 (table test-table [id=98bd3da96c544131a122138977f40d16]) successfully deleted
W20260923 15:14:01.855005 4334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:01.855229 4334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:01.855265 4334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:01.856439 4334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:01.856518 4334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:01.856563 4334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:14:01.858035 4334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:46227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=34705
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:01.858299 4334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:01.858525 4334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:01.863425 4345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.863451 4344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.863432 4347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:01.863895 4334 server_base.cc:1034] running on GCE node
I20260923 15:14:01.864104 4334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:01.864332 4334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:01.865504 4334 hybrid_clock.cc:648] HybridClock initialized: now 1790176441865496 us; error 35 us; skew 500 ppm
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:01.867995 4334 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:14:01.868376 4334 webserver.cc:460] Webserver started at http://127.21.204.66:34705/ using document root <none> and password file <none>
I20260923 15:14:01.868568 4334 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:01.868618 4334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:01.869836 4334 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:01.870466 4354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.870635 4334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.870720 4334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "bf92d8c74f4d4b4194de19b158da0316"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "3daed68ff2174b1d9a1e4ee347411582"
server_key_iv: "e7d14495a9a19c6bc03d59a7898501ba"
server_key_version: "encryptionkey@0"
I20260923 15:14:01.871039 4334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:01.890000 4334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:01.892081 4334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:01.892225 4334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:01.892468 4334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:01.892776 4334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:01.892831 4334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.892875 4334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:01.892903 4334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:01.899518 4334 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:46227
I20260923 15:14:01.899595 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:46227 every 8 connection(s)
I20260923 15:14:01.899942 4334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:14:01.903259 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 4334
I20260923 15:14:01.903527 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:36281
--local_ip_for_outbound_sockets=127.21.204.67
--tserver_master_addrs=127.21.204.126:41409
--webserver_port=41997
--webserver_interface=127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:01.910881 4333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.900680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:46121 (local address 127.21.204.126:41409)
0923 15:14:01.900745 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:01.900748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:01.900954 (+ 206us) server_negotiation.cc:407] Connection header received
0923 15:14:01.901489 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:01.901490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:01.901503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:01.901552 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:01.902575 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.903108 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.904243 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.904463 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.904555 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.906895 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:01.906897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:01.906898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:01.906906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:01.909108 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.909593 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.909594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.909595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.909663 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:01.910128 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:01.910129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:01.910130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:01.910321 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:01.910412 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:01.910695 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 15:14:01.910791 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:14:01.911234 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:01.900682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:46121)
0923 15:14:01.900833 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:01.900868 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:01.901381 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:01.901617 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:01.901620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:01.902266 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:01.902450 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.902454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.903211 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.903214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.904146 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:01.904149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:01.904570 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:01.904571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:01.904694 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:01.905203 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:01.905212 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:01.906786 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:01.909264 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.909275 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.909282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.909512 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.909804 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:01.909806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:01.909806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:01.910019 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:01.910406 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:01.910413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:01.910555 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 15:14:01.910658 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 15:14:01.910824 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260923 15:14:01.911775 4468 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41409
I20260923 15:14:01.911906 4468 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:01.912129 4468 heartbeater.cc:511] Master 127.21.204.126:41409 requested a full tablet report, sending...
I20260923 15:14:01.912599 4142 ts_manager.cc:194] Registered new tserver with Master: bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
I20260923 15:14:01.913164 4142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:46121
W20260923 15:14:01.988742 4471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:01.988930 4471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:01.988955 4471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:01.990136 4471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:01.990180 4471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:01.990206 4471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:14:01.991602 4471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:36281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=41997
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:01.991780 4471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:01.991951 4471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:01.996976 4476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:01.996961 4479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:01.997107 4471 server_base.cc:1034] running on GCE node
W20260923 15:14:01.996958 4477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:01.997349 4471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:01.997596 4471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:01.998768 4471 hybrid_clock.cc:648] HybridClock initialized: now 1790176441998745 us; error 36 us; skew 500 ppm
Sep 23 15:14:01 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:02.001365 4471 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:14:02.001850 4471 webserver.cc:460] Webserver started at http://127.21.204.67:41997/ using document root <none> and password file <none>
I20260923 15:14:02.002151 4471 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:02.002211 4471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:02.003527 4471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.004209 4486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.004434 4471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.004534 4471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "a4651b86ef91410498814075e11c3595"
format_stamp: "Formatted at 2026-09-23 15:14:01 on dist-test-slave-t5lt"
server_key: "6b6a74290e789a138fe39183eeb19b33"
server_key_iv: "c40ae4ac7ddc6f8f8037b23638f91d30"
server_key_version: "encryptionkey@0"
I20260923 15:14:02.004861 4471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:02.054843 4471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:02.056978 4471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:02.057165 4471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:02.057420 4471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:02.057735 4471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:02.057786 4471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.057827 4471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:02.057866 4471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.063843 4471 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:36281
I20260923 15:14:02.063889 4599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:36281 every 8 connection(s)
I20260923 15:14:02.064302 4471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:14:02.069422 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 4471
Sep 23 15:14:02 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176441, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:02.073828 4333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.065036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:40105 (local address 127.21.204.126:41409)
0923 15:14:02.065093 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.065096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.065356 (+ 260us) server_negotiation.cc:407] Connection header received
0923 15:14:02.065815 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.065816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.065827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.065910 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:02.066826 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.067320 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.068313 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.068534 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.068624 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.070863 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:02.070865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:02.070865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:02.070871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:02.072436 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:02.072887 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:02.072888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:02.072889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:02.072943 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:02.073205 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:02.073206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:02.073207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:02.073318 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:02.073394 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.073655 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.073733 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:14:02.074105 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.065039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:40105)
0923 15:14:02.065228 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.065264 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:02.065727 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:02.065988 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:02.065997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:02.066550 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:02.066738 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.066742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.067410 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.067411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.068223 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.068225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.068636 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.068637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.068752 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.069175 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:02.069183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:02.070757 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:02.072538 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:02.072546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:02.072549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:02.072816 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:02.073018 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:02.073019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:02.073019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:02.073137 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:02.073386 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:02.073393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:02.073526 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 15:14:02.073632 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 15:14:02.073838 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260923 15:14:02.074530 4600 heartbeater.cc:347] Connected to a master server at 127.21.204.126:41409
I20260923 15:14:02.074656 4600 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:02.074839 4600 heartbeater.cc:511] Master 127.21.204.126:41409 requested a full tablet report, sending...
I20260923 15:14:02.075273 4142 ts_manager.cc:194] Registered new tserver with Master: a4651b86ef91410498814075e11c3595 (127.21.204.67:36281)
I20260923 15:14:02.075632 4142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:40105
I20260923 15:14:02.076574 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.069857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.21.204.126:41409)
0923 15:14:02.070005 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.070007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.070018 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:14:02.070043 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.070044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.070080 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.070157 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:02.070760 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.071421 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.072228 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.072445 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.072530 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.073690 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:02.073691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:02.073692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:02.073699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:02.075454 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:02.075813 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:02.075814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:02.075814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:02.075875 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:02.076074 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:02.076075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:02.076076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:02.076194 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:02.076366 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.076406 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.076485 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260923 15:14:02.077265 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:14:02.079754 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.077659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.21.204.126:41409)
0923 15:14:02.077692 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.077694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.077725 (+ 31us) server_negotiation.cc:407] Connection header received
0923 15:14:02.077784 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.077785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.077812 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.077929 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:14:02.078013 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.078428 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.079107 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.079288 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.079340 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.079496 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:14:02.079593 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:14:02.079676 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.079691 (+ 15us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.079710 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 15:14:02.080042 4142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:14:02.081619 4142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:14:02.083904 4264 tablet_service.cc:1467] Processing CreateTablet for tablet 6698010813d64dc8baf14832b7cdd68d (DEFAULT_TABLE table=test-table [id=5e28234c0457400bbb36aaada6afcc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:02.084120 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6698010813d64dc8baf14832b7cdd68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:02.085541 4222 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap starting.
I20260923 15:14:02.085898 4222 tablet_bootstrap.cc:654] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:02.086931 4222 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: No bootstrap required, opened a new log
I20260923 15:14:02.086983 4222 ts_tablet_manager.cc:1397] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.087128 4222 raft_consensus.cc:348] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.087216 4222 raft_consensus.cc:374] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:02.087268 4222 raft_consensus.cc:729] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa7cdceafab4fd6904356d00bdd53d7, State: Initialized, Role: FOLLOWER
I20260923 15:14:02.087316 4222 consensus_queue.cc:260] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.087417 4222 ts_tablet_manager.cc:1428] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 15:14:02.087437 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.083420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:36281 (local address 127.0.0.1:34452)
0923 15:14:02.083474 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.083484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:02.083521 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:02.083891 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:02.083891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:02.083893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:02.083926 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.083927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.085090 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.085090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.086073 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.086076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.087120 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.087121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.087193 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.087211 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 15:14:02.087275 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 15:14:02.087369 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":36}
I20260923 15:14:02.087572 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.083501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.21.204.67:36281)
0923 15:14:02.083671 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.083676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.083687 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:14:02.083723 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.083725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.083740 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.083861 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:02.083975 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.084967 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.086151 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.086989 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.087099 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.087163 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.087439 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.087489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260923 15:14:02.088253 4534 tablet_service.cc:1467] Processing CreateTablet for tablet 6698010813d64dc8baf14832b7cdd68d (DEFAULT_TABLE table=test-table [id=5e28234c0457400bbb36aaada6afcc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:02.088609 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6698010813d64dc8baf14832b7cdd68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:02.089473 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.083536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46227 (local address 127.0.0.1:56576)
0923 15:14:02.083713 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.083723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:02.083762 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:02.085923 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:02.085924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:02.085926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:02.085971 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.085974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.087210 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.087211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.088376 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.088379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.089219 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.089220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.089284 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.089288 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:14:02.089366 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 15:14:02.089405 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260923 15:14:02.089500 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.083648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.21.204.66:46227)
0923 15:14:02.085629 (+ 1981us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.085633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.085644 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:14:02.085689 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.085690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.085707 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.085788 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:02.086080 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.087090 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.088472 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.089122 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.089201 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.089253 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.089384 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.089422 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":93,"threads_started":1}
I20260923 15:14:02.090132 4400 tablet_service.cc:1467] Processing CreateTablet for tablet 6698010813d64dc8baf14832b7cdd68d (DEFAULT_TABLE table=test-table [id=5e28234c0457400bbb36aaada6afcc4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:02.090399 4400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6698010813d64dc8baf14832b7cdd68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:02.091620 4610 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Bootstrap starting.
I20260923 15:14:02.092247 4610 tablet_bootstrap.cc:654] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:02.092741 4610 log.cc:824] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:02.093146 4612 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Bootstrap starting.
I20260923 15:14:02.093609 4610 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: No bootstrap required, opened a new log
I20260923 15:14:02.093683 4610 ts_tablet_manager.cc:1397] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:14:02.093806 4612 tablet_bootstrap.cc:654] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:02.094424 4612 log.cc:824] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:02.094771 4336 raft_consensus.cc:482] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:02.094877 4336 raft_consensus.cc:504] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.095175 4336 leader_election.cc:288] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4651b86ef91410498814075e11c3595 (127.21.204.67:36281), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
I20260923 15:14:02.095201 4612 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: No bootstrap required, opened a new log
I20260923 15:14:02.095263 4612 ts_tablet_manager.cc:1397] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 15:14:02.096014 4610 raft_consensus.cc:348] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.096181 4610 raft_consensus.cc:374] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:02.096222 4610 raft_consensus.cc:729] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4651b86ef91410498814075e11c3595, State: Initialized, Role: FOLLOWER
I20260923 15:14:02.096351 4610 consensus_queue.cc:260] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.096604 4610 ts_tablet_manager.cc:1428] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:14:02.096714 4600 heartbeater.cc:503] Master 127.21.204.126:41409 was elected leader, sending a full tablet report...
I20260923 15:14:02.097445 4612 raft_consensus.cc:348] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.097620 4612 raft_consensus.cc:374] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:02.097663 4612 raft_consensus.cc:729] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf92d8c74f4d4b4194de19b158da0316, State: Initialized, Role: FOLLOWER
I20260923 15:14:02.097777 4612 consensus_queue.cc:260] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.098052 4612 ts_tablet_manager.cc:1428] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 15:14:02.098187 4468 heartbeater.cc:503] Master 127.21.204.126:41409 was elected leader, sending a full tablet report...
I20260923 15:14:02.098989 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.095669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:47897 (local address 127.21.204.67:36281)
0923 15:14:02.095755 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.095757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.095765 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:14:02.095786 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.095787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.095795 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.095847 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:02.096023 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.096684 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.098094 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.098606 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.098673 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.098682 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.098895 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.098929 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 15:14:02.099007 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.095324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:36281 (local address 127.21.204.65:47897)
0923 15:14:02.095368 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.095380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:02.095432 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:02.095873 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:02.095875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:02.095877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:02.095940 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.095941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.096785 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.096786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.097986 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.097989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.098684 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.098685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.098774 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.098796 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 15:14:02.098885 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 15:14:02.098929 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":3}
I20260923 15:14:02.099291 4554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6698010813d64dc8baf14832b7cdd68d" candidate_uuid: "daa7cdceafab4fd6904356d00bdd53d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4651b86ef91410498814075e11c3595" is_pre_election: true
I20260923 15:14:02.099433 4554 raft_consensus.cc:2454] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daa7cdceafab4fd6904356d00bdd53d7 in term 0.
I20260923 15:14:02.099613 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.095434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46227 (local address 127.21.204.65:51035)
0923 15:14:02.095985 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.095996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:02.096047 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:02.096348 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:02.096349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:02.096351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:02.096417 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.096468 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.097406 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.097407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.098599 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:02.098602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.099326 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.099327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:02.099413 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.099418 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:14:02.099477 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:14:02.099515 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":4,"thread_start_us":94,"threads_started":1}
I20260923 15:14:02.099679 4218 leader_election.cc:302] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4651b86ef91410498814075e11c3595, daa7cdceafab4fd6904356d00bdd53d7; no voters:
I20260923 15:14:02.099858 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.095708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:51035 (local address 127.21.204.66:46227)
0923 15:14:02.095775 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.095777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.096111 (+ 334us) server_negotiation.cc:407] Connection header received
0923 15:14:02.096132 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.096132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.096141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.096305 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:02.096543 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.097309 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.098710 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.099236 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.099300 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.099305 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.099690 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.099786 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 15:14:02.099855 4336 raft_consensus.cc:2781] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:14:02.099917 4336 raft_consensus.cc:482] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:14:02.099948 4336 raft_consensus.cc:3037] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:02.100584 4336 raft_consensus.cc:504] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.100721 4336 leader_election.cc:288] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 1 election: Requested vote from peers a4651b86ef91410498814075e11c3595 (127.21.204.67:36281), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
I20260923 15:14:02.100879 4422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6698010813d64dc8baf14832b7cdd68d" candidate_uuid: "daa7cdceafab4fd6904356d00bdd53d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf92d8c74f4d4b4194de19b158da0316" is_pre_election: true
I20260923 15:14:02.100885 4554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6698010813d64dc8baf14832b7cdd68d" candidate_uuid: "daa7cdceafab4fd6904356d00bdd53d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4651b86ef91410498814075e11c3595"
I20260923 15:14:02.100982 4422 raft_consensus.cc:2454] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daa7cdceafab4fd6904356d00bdd53d7 in term 0.
I20260923 15:14:02.100960 4554 raft_consensus.cc:3037] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:02.101104 4421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6698010813d64dc8baf14832b7cdd68d" candidate_uuid: "daa7cdceafab4fd6904356d00bdd53d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf92d8c74f4d4b4194de19b158da0316"
I20260923 15:14:02.101186 4421 raft_consensus.cc:3037] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:02.101691 4554 raft_consensus.cc:2454] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa7cdceafab4fd6904356d00bdd53d7 in term 1.
I20260923 15:14:02.101760 4421 raft_consensus.cc:2454] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa7cdceafab4fd6904356d00bdd53d7 in term 1.
I20260923 15:14:02.101857 4218 leader_election.cc:302] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4651b86ef91410498814075e11c3595, daa7cdceafab4fd6904356d00bdd53d7; no voters:
I20260923 15:14:02.102007 4336 raft_consensus.cc:2781] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:14:02.102156 4336 raft_consensus.cc:686] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 LEADER]: Becoming Leader. State: Replica: daa7cdceafab4fd6904356d00bdd53d7, State: Running, Role: LEADER
I20260923 15:14:02.102224 4336 consensus_queue.cc:237] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.102926 4141 catalog_manager.cc:5495] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65). New cstate: current_term: 1 leader_uuid: "daa7cdceafab4fd6904356d00bdd53d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } health_report { overall_health: HEALTHY } } }
I20260923 15:14:02.109836 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.107115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.21.204.65:44879)
0923 15:14:02.107180 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:02.107182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:02.107192 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:14:02.107252 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:02.107253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:02.107272 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:02.107345 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:14:02.107511 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.108072 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.108972 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:02.109212 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:02.109313 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:02.109482 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:14:02.109579 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:14:02.109668 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:02.109735 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 15:14:02.109757 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:14:02.111549 4554 raft_consensus.cc:1261] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 1 FOLLOWER]: Refusing update from remote peer daa7cdceafab4fd6904356d00bdd53d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:02.111557 4421 raft_consensus.cc:1261] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Refusing update from remote peer daa7cdceafab4fd6904356d00bdd53d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:02.111860 4617 consensus_queue.cc:1035] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:02.112016 4336 consensus_queue.cc:1035] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:02.114817 4621 mvcc.cc:205] Tried to move back new op lower bound from 7332562706885505024 to 7332562706851274752. Current Snapshot: MvccSnapshot[applied={T|T < 7332562706885505024}]
I20260923 15:14:02.115021 4620 mvcc.cc:205] Tried to move back new op lower bound from 7332562706885505024 to 7332562706851274752. Current Snapshot: MvccSnapshot[applied={T|T < 7332562706885505024 or (T in {7332562706885505024})}]
I20260923 15:14:02.115046 4622 mvcc.cc:205] Tried to move back new op lower bound from 7332562706885505024 to 7332562706851274752. Current Snapshot: MvccSnapshot[applied={T|T < 7332562706885505024}]
I20260923 15:14:02.116590 4142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:02.116688 4142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:02.117372 4142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Sending DeleteTablet for 3 replicas of tablet 6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:02.117728 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet 6698010813d64dc8baf14832b7cdd68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:02 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:37264
I20260923 15:14:02.117707 4534 tablet_service.cc:1514] Processing DeleteTablet for tablet 6698010813d64dc8baf14832b7cdd68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:02 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:34452
I20260923 15:14:02.117769 4400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6698010813d64dc8baf14832b7cdd68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:02 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:56576
I20260923 15:14:02.117810 4339 tablet_replica.cc:330] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: stopping tablet replica
I20260923 15:14:02.117858 4339 raft_consensus.cc:2229] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:14:02.118034 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 4181
I20260923 15:14:02.118069 4339 raft_consensus.cc:2258] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:02.119264 4630 tablet_replica.cc:330] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: stopping tablet replica
I20260923 15:14:02.119383 4630 raft_consensus.cc:2229] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:02.119481 4630 raft_consensus.cc:2258] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:02.119879 4630 ts_tablet_manager.cc:1905] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:02.121363 4630 ts_tablet_manager.cc:1918] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:02.121452 4630 log.cc:1196] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:02.121717 4630 ts_tablet_manager.cc:1939] T 6698010813d64dc8baf14832b7cdd68d P a4651b86ef91410498814075e11c3595: Deleting consensus metadata
I20260923 15:14:02.121989 4631 tablet_replica.cc:330] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: stopping tablet replica
I20260923 15:14:02.122107 4631 raft_consensus.cc:2229] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:02.122165 4128 catalog_manager.cc:4841] TS a4651b86ef91410498814075e11c3595 (127.21.204.67:36281): tablet 6698010813d64dc8baf14832b7cdd68d (table test-table [id=5e28234c0457400bbb36aaada6afcc4c]) successfully deleted
I20260923 15:14:02.122220 4631 raft_consensus.cc:2258] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:02.122604 4631 ts_tablet_manager.cc:1905] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 15:14:02.123682 4128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:44879 (error 108)
I20260923 15:14:02.123957 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 4334
I20260923 15:14:02.124063 4631 ts_tablet_manager.cc:1918] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:02.124130 4631 log.cc:1196] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:02.124172 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.124050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:44879 (local address 127.0.0.1:37284)
0923 15:14:02.124098 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.124132 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 15:14:02.124279 4128 catalog_manager.cc:4584] TS daa7cdceafab4fd6904356d00bdd53d7 (127.21.204.65:44879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6698010813d64dc8baf14832b7cdd68d: Network error: Client connection negotiation failed: client connection to 127.21.204.65:44879: connect: Connection refused (error 111)
I20260923 15:14:02.129693 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 4471
I20260923 15:14:02.129745 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.129622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46227 (local address 127.0.0.1:56592)
0923 15:14:02.129677 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.129719 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.66:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 15:14:02.134450 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 4109
I20260923 15:14:02.139616 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41409
--webserver_interface=127.21.204.126
--webserver_port=34711
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:02.219875 4633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:02.220088 4633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:02.220111 4633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:02.221276 4633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:14:02.221318 4633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:02.221333 4633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:02.221345 4633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:14:02.221398 4633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:14:02.222891 4633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=34711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:02.223155 4633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:02.223414 4633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:02.228735 4638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.228744 4639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.228794 4641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:02.229038 4633 server_base.cc:1034] running on GCE node
I20260923 15:14:02.229259 4633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:02.229491 4633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:02.230628 4633 hybrid_clock.cc:648] HybridClock initialized: now 1790176442230603 us; error 38 us; skew 500 ppm
Sep 23 15:14:02 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:02.232918 4633 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:14:02.233363 4633 webserver.cc:460] Webserver started at http://127.21.204.126:34711/ using document root <none> and password file <none>
I20260923 15:14:02.233606 4633 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:02.233655 4633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:02.235054 4633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.235802 4648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.235999 4633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.236152 4633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "916928e1a565483a93532f4dafdc84b0"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "9a928352092cf53e01f6b2b3ede9dea0"
server_key_iv: "87c520aeb353612d84ea355f06f5b284"
server_key_version: "encryptionkey@0"
I20260923 15:14:02.236451 4633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:02.258093 4633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:02.260004 4633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:02.260119 4633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:02.264321 4633 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:41409
I20260923 15:14:02.264423 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:41409 every 8 connection(s)
I20260923 15:14:02.264794 4633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:14:02.266469 4701 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:14:02.267105 4701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Bootstrap starting.
I20260923 15:14:02.274788 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 4633
I20260923 15:14:02.275141 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:44879
--local_ip_for_outbound_sockets=127.21.204.65
--tserver_master_addrs=127.21.204.126:41409
--webserver_port=36119
--webserver_interface=127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:14:02.275740 4701 log.cc:824] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:02.278200 4701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:14:02.278488 4701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0: Bootstrap complete.
I20260923 15:14:02.280717 4701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:02.280932 4701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916928e1a565483a93532f4dafdc84b0, State: Initialized, Role: FOLLOWER
I20260923 15:14:02.281103 4701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:02.281164 4701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:14:02.281239 4701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:14:02.281286 4701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 2 FOLLOWER]: Advancing to term 3
I20260923 15:14:02.282369 4701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:02.282460 4701 leader_election.cc:302] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916928e1a565483a93532f4dafdc84b0; no voters:
I20260923 15:14:02.282645 4701 leader_election.cc:288] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 15:14:02.282691 4705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 15:14:02.282807 4705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [term 3 LEADER]: Becoming Leader. State: Replica: 916928e1a565483a93532f4dafdc84b0, State: Running, Role: LEADER
I20260923 15:14:02.282892 4705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } }
I20260923 15:14:02.282918 4701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:14:02.283174 4705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916928e1a565483a93532f4dafdc84b0. Latest consensus state: current_term: 3 leader_uuid: "916928e1a565483a93532f4dafdc84b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } } }
I20260923 15:14:02.283250 4705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:02.283179 4706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "916928e1a565483a93532f4dafdc84b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916928e1a565483a93532f4dafdc84b0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41409 } } }
I20260923 15:14:02.283299 4706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916928e1a565483a93532f4dafdc84b0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:02.283859 4709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:14:02.284711 4709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:14:02.284832 4709 catalog_manager.cc:1259] Loaded cluster ID: c80f993f08094b03b1e92fca59511c41
I20260923 15:14:02.284961 4709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 15:14:02.285267 4709 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe5913899ad google::LogMessage::Fail() at ??:0
@ 0x7fe59138c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe5913894de google::LogMessage::Flush() at ??:0
@ 0x7fe59138aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5992aa0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe5992b38b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe5932570d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe59324c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe5940b06db start_thread at ??:0
@ 0x7fe58fe8761f clone at ??:0
W20260923 15:14:02.392009 4704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:02.392302 4704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:02.392344 4704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:02.394119 4704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:02.394179 4704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:02.394220 4704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:14:02.396342 4704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:44879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=36119
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:02.396589 4704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:02.396829 4704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:02.403729 4737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.403733 4740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.403771 4738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:02.403980 4704 server_base.cc:1034] running on GCE node
I20260923 15:14:02.404258 4704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:02.404501 4704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:02.405675 4704 hybrid_clock.cc:648] HybridClock initialized: now 1790176442405645 us; error 49 us; skew 500 ppm
Sep 23 15:14:02 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:02.408318 4704 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:14:02.408849 4704 webserver.cc:460] Webserver started at http://127.21.204.65:36119/ using document root <none> and password file <none>
I20260923 15:14:02.409107 4704 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:02.409188 4704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:02.410753 4704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.411502 4747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.411724 4704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.411840 4704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "daa7cdceafab4fd6904356d00bdd53d7"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "66f9dd56c562f8be7f553f8cc4501407"
server_key_iv: "95992d4af8d79b28f92ada8c5280446e"
server_key_version: "encryptionkey@0"
I20260923 15:14:02.412209 4704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:02.424500 4704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:02.426733 4704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:02.426900 4704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:02.427143 4704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:02.427609 4754 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 15:14:02.429401 4704 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 15:14:02.429466 4704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 15:14:02.429541 4704 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 15:14:02.430270 4704 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 15:14:02.430331 4704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.430390 4754 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap starting.
I20260923 15:14:02.438308 4704 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:44879
I20260923 15:14:02.438457 4861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:44879 every 8 connection(s)
I20260923 15:14:02.438767 4704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:14:02.441118 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 4704
I20260923 15:14:02.441246 4754 log.cc:824] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:02.441468 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:46227
--local_ip_for_outbound_sockets=127.21.204.66
--tserver_master_addrs=127.21.204.126:41409
--webserver_port=34705
--webserver_interface=127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:14:02.443943 4754 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:14:02.444269 4754 tablet_bootstrap.cc:492] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Bootstrap complete.
I20260923 15:14:02.445605 4754 ts_tablet_manager.cc:1397] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260923 15:14:02.447119 4754 raft_consensus.cc:348] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.447458 4754 raft_consensus.cc:729] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa7cdceafab4fd6904356d00bdd53d7, State: Initialized, Role: FOLLOWER
I20260923 15:14:02.447629 4754 consensus_queue.cc:260] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:02.447912 4754 ts_tablet_manager.cc:1428] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260923 15:14:02.525337 4866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:02.525583 4866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:02.525624 4866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:02.526957 4866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:02.527032 4866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:02.527081 4866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:14:02.528690 4866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:46227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=34705
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:02.528973 4866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:02.529227 4866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:02.534409 4872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.534375 4875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.534385 4873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:02.534596 4866 server_base.cc:1034] running on GCE node
I20260923 15:14:02.534875 4866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:02.535151 4866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:02.536330 4866 hybrid_clock.cc:648] HybridClock initialized: now 1790176442536306 us; error 38 us; skew 500 ppm
Sep 23 15:14:02 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:02.539456 4866 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:14:02.540004 4866 webserver.cc:460] Webserver started at http://127.21.204.66:34705/ using document root <none> and password file <none>
I20260923 15:14:02.540261 4866 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:02.540333 4866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260923 15:14:02.540582 4864 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:14:02.439479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:51295)
0923 15:14:02.439684 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.439731 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:02.440204 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:02.540420 (+100216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":114,"threads_started":1}
W20260923 15:14:02.540840 4862 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:41409 (0 consecutive failures): Network error: Failed to ping master at 127.21.204.126:41409: Client connection negotiation failed: client connection to 127.21.204.126:41409: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 15:14:02.541168 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.541048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:53059)
0923 15:14:02.541102 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.541150 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 15:14:02.541615 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.541483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:40075)
0923 15:14:02.541561 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.541594 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 15:14:02.541791 4862 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 15:14:02.542114 4866 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:14:02.542742 4882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.542924 4866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.543052 4866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "bf92d8c74f4d4b4194de19b158da0316"
format_stamp: "Formatted at 2026-09-23 15:14:00 on dist-test-slave-t5lt"
server_key: "3daed68ff2174b1d9a1e4ee347411582"
server_key_iv: "e7d14495a9a19c6bc03d59a7898501ba"
server_key_version: "encryptionkey@0"
I20260923 15:14:02.543386 4866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:02.560873 4866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:02.562892 4866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:02.563037 4866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:02.563287 4866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:02.563747 4889 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260923 15:14:02.565224 4889 ts_tablet_manager.cc:1852] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 15:14:02.565487 4889 ts_tablet_manager.cc:1905] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:02.567553 4889 ts_tablet_manager.cc:1918] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:02.567634 4889 log.cc:1196] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:02.567878 4889 ts_tablet_manager.cc:1939] T 6698010813d64dc8baf14832b7cdd68d P bf92d8c74f4d4b4194de19b158da0316: Deleting consensus metadata
I20260923 15:14:02.568089 4866 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 15:14:02.568151 4866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 15:14:02.568195 4866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:02.568216 4866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.574656 4866 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:46227
I20260923 15:14:02.574719 4997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:46227 every 8 connection(s)
I20260923 15:14:02.574993 4866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:14:02.576013 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.575606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:49441)
0923 15:14:02.575792 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.575872 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
W20260923 15:14:02.576241 4998 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:41409 (0 consecutive failures): Network error: Failed to ping master at 127.21.204.126:41409: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
I20260923 15:14:02.576520 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.576418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:46445)
0923 15:14:02.576469 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.576494 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 15:14:02.576797 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.576712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:45581)
0923 15:14:02.576756 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.576777 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 15:14:02.576893 4998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 15:14:02.578393 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 4866
I20260923 15:14:02.578622 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:36281
--local_ip_for_outbound_sockets=127.21.204.67
--tserver_master_addrs=127.21.204.126:41409
--webserver_port=41997
--webserver_interface=127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:02.660151 5001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:02.660352 5001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:02.660375 5001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:02.661507 5001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:02.661548 5001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:02.661574 5001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:14:02.663426 5001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:36281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=41997
--tserver_master_addrs=127.21.204.126:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:02.663726 5001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:02.664026 5001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:02.668901 5007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.668946 5006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:02.668929 5009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:02.669135 5001 server_base.cc:1034] running on GCE node
I20260923 15:14:02.669306 5001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:02.669489 5001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:02.670625 5001 hybrid_clock.cc:648] HybridClock initialized: now 1790176442670613 us; error 25 us; skew 500 ppm
Sep 23 15:14:02 dist-test-slave-t5lt krb5kdc[3561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:02.673043 5001 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:14:02.673473 5001 webserver.cc:460] Webserver started at http://127.21.204.67:41997/ using document root <none> and password file <none>
I20260923 15:14:02.673707 5001 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:02.673753 5001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:02.675133 5001 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.675824 5016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.676069 5001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:02.676195 5001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "a4651b86ef91410498814075e11c3595"
format_stamp: "Formatted at 2026-09-23 15:14:01 on dist-test-slave-t5lt"
server_key: "6b6a74290e789a138fe39183eeb19b33"
server_key_iv: "c40ae4ac7ddc6f8f8037b23638f91d30"
server_key_version: "encryptionkey@0"
I20260923 15:14:02.676523 5001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:02.712548 5001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:02.715005 5001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:02.715181 5001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:02.715431 5001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:02.715751 5001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:02.715806 5001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.715840 5001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:02.715867 5001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:02.721473 5001 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:36281
I20260923 15:14:02.721565 5131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:36281 every 8 connection(s)
I20260923 15:14:02.721920 5001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:14:02.722931 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.722561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:40569)
0923 15:14:02.722717 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.722799 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
W20260923 15:14:02.723356 5132 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:41409 (0 consecutive failures): Network error: Failed to ping master at 127.21.204.126:41409: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
I20260923 15:14:02.723625 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5001
I20260923 15:14:02.723727 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.723542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:34803)
0923 15:14:02.723640 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.723670 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 15:14:02.724313 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:02.724201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:43597)
0923 15:14:02.724255 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:02.724281 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 15:14:02.724421 5132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 15:14:03.542438 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:03.542152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:37221)
0923 15:14:03.542305 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:03.542354 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260923 15:14:03.577447 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:03.577225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:41599)
0923 15:14:03.577377 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:03.577412 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260923 15:14:03.724973 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:03.724757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:37653)
0923 15:14:03.724886 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:03.724921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260923 15:14:03.782825 5139 raft_consensus.cc:482] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:03.783010 5139 raft_consensus.cc:504] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:03.783401 5139 leader_election.cc:288] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4651b86ef91410498814075e11c3595 (127.21.204.67:36281), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
Sep 23 15:14:03 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.67@KRBTEST.COM
Sep 23 15:14:03 dist-test-slave-t5lt krb5kdc[3561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.66@KRBTEST.COM
I20260923 15:14:03.794956 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:03.783744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:58445 (local address 127.21.204.67:36281)
0923 15:14:03.784067 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:03.784077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:03.784118 (+ 41us) server_negotiation.cc:407] Connection header received
0923 15:14:03.784695 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:03.784697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:03.784741 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:03.784856 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:03.785697 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.786736 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.787686 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.788256 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.788420 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:03.790994 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:03.790999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:03.790999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:03.791013 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:03.793048 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:03.793532 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:03.793535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:03.793536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:03.793661 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:03.794061 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:03.794062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:03.794062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:03.794291 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:03.794406 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:03.794685 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 15:14:03.794829 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260923 15:14:03.794950 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:03.783758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:46227 (local address 127.21.204.65:51181)
0923 15:14:03.783868 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:03.783880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:03.784116 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:03.784933 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:03.784934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:03.785360 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:03.785540 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:03.785546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.786709 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.786710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:03.787610 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:03.787613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.788402 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.788404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:03.788499 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:03.789112 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:03.789120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:03.791158 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:03.793251 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:03.793252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:03.793253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:03.793515 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:03.793801 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:03.793802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:03.793802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:03.793917 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:03.794344 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:03.794352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:03.794493 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 15:14:03.794615 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 15:14:03.794818 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":53,"thread_start_us":62,"threads_started":1}
I20260923 15:14:03.795054 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:03.783591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:36281 (local address 127.21.204.65:58445)
0923 15:14:03.783707 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:03.783725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:03.783979 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:03.784926 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:03.784930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:03.785360 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:03.785540 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:03.785546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.786842 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.786843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:03.787594 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:03.787596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.788380 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.788381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:03.788508 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:03.789163 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:03.789168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:03.790776 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:03.793147 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:03.793156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:03.793158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:03.793446 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:03.793737 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:03.793738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:03.793738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:03.793872 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:03.794405 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:03.794406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:03.794496 (+ 90us) client_negotiation.cc:769] Sending connection context
0923 15:14:03.794616 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 15:14:03.794914 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":82}
I20260923 15:14:03.795445 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:03.783844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:51181 (local address 127.21.204.66:46227)
0923 15:14:03.783988 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:03.784001 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:03.784119 (+ 118us) server_negotiation.cc:407] Connection header received
0923 15:14:03.784739 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:03.784742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:03.784782 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:03.784934 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:03.785666 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.786592 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.787737 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:03.788255 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:03.788420 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:03.791271 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:03.791274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:03.791275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:03.791287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:03.793157 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:03.793605 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:03.793608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:03.793608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:03.793719 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:03.793995 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:03.793996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:03.793997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:03.794265 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:03.794331 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:03.795170 (+ 839us) server_negotiation.cc:299] Negotiation successful
0923 15:14:03.795343 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
W20260923 15:14:03.795899 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4651b86ef91410498814075e11c3595 (127.21.204.67:36281): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
W20260923 15:14:03.796293 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:03.796379 4750 leader_election.cc:302] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daa7cdceafab4fd6904356d00bdd53d7; no voters: a4651b86ef91410498814075e11c3595, bf92d8c74f4d4b4194de19b158da0316
I20260923 15:14:03.796500 5139 raft_consensus.cc:2726] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 15:14:04.543262 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:04.543013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:60617)
0923 15:14:04.543173 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:04.543213 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260923 15:14:04.578148 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:04.577963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:42247)
0923 15:14:04.578078 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:04.578113 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260923 15:14:04.725710 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:04.725507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:55039)
0923 15:14:04.725638 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:04.725675 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260923 15:14:05.492215 5146 raft_consensus.cc:482] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:05.492316 5146 raft_consensus.cc:504] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:05.492514 5146 leader_election.cc:288] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4651b86ef91410498814075e11c3595 (127.21.204.67:36281), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
W20260923 15:14:05.493014 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
W20260923 15:14:05.493113 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4651b86ef91410498814075e11c3595 (127.21.204.67:36281): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:05.493149 4750 leader_election.cc:302] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daa7cdceafab4fd6904356d00bdd53d7; no voters: a4651b86ef91410498814075e11c3595, bf92d8c74f4d4b4194de19b158da0316
I20260923 15:14:05.493265 5146 raft_consensus.cc:2726] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 15:14:05.544040 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:05.543812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:54423)
0923 15:14:05.543954 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:05.543989 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260923 15:14:05.578893 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:05.578648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:36503)
0923 15:14:05.578805 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:05.578854 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260923 15:14:05.726429 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:05.726224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:50837)
0923 15:14:05.726354 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:05.726390 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260923 15:14:06.544728 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:06.544512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:49217)
0923 15:14:06.544646 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:06.544682 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260923 15:14:06.579469 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:06.579286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:59245)
0923 15:14:06.579399 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:06.579432 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260923 15:14:06.727080 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:06.726874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:34831)
0923 15:14:06.727006 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:06.727041 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260923 15:14:07.545413 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:07.545200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:41595)
0923 15:14:07.545329 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:07.545363 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260923 15:14:07.580127 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:07.579927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:36743)
0923 15:14:07.580048 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:07.580087 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260923 15:14:07.727805 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:07.727570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:33839)
0923 15:14:07.727726 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:07.727764 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260923 15:14:07.753983 5156 raft_consensus.cc:482] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:07.754079 5156 raft_consensus.cc:504] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:07.754280 5156 leader_election.cc:288] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4651b86ef91410498814075e11c3595 (127.21.204.67:36281), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
W20260923 15:14:07.754703 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4651b86ef91410498814075e11c3595 (127.21.204.67:36281): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
W20260923 15:14:07.754818 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:07.754855 4750 leader_election.cc:302] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daa7cdceafab4fd6904356d00bdd53d7; no voters: a4651b86ef91410498814075e11c3595, bf92d8c74f4d4b4194de19b158da0316
I20260923 15:14:07.754940 5156 raft_consensus.cc:2726] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 15:14:08.546173 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:08.545866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:32797)
0923 15:14:08.546085 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:08.546125 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":161,"threads_started":1}
I20260923 15:14:08.580806 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:08.580595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:42147)
0923 15:14:08.580738 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:08.580767 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260923 15:14:08.728622 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:08.728358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:38623)
0923 15:14:08.728536 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:08.728576 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260923 15:14:09.546903 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:09.546646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:50855)
0923 15:14:09.546819 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:09.546855 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260923 15:14:09.581490 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:09.581284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:59647)
0923 15:14:09.581424 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:09.581454 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260923 15:14:09.729413 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:09.729165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:37135)
0923 15:14:09.729319 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:09.729366 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260923 15:14:10.547550 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:10.547360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:49691)
0923 15:14:10.547472 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:10.547504 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260923 15:14:10.582296 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:10.582075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:40779)
0923 15:14:10.582223 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:10.582254 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260923 15:14:10.730266 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:10.730072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:37151)
0923 15:14:10.730193 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:10.730223 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260923 15:14:11.413038 5166 raft_consensus.cc:482] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:11.413127 5166 raft_consensus.cc:504] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:11.413335 5166 leader_election.cc:288] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4651b86ef91410498814075e11c3595 (127.21.204.67:36281), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
W20260923 15:14:11.413797 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4651b86ef91410498814075e11c3595 (127.21.204.67:36281): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
W20260923 15:14:11.413930 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:11.413970 4750 leader_election.cc:302] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daa7cdceafab4fd6904356d00bdd53d7; no voters: a4651b86ef91410498814075e11c3595, bf92d8c74f4d4b4194de19b158da0316
I20260923 15:14:11.414096 5166 raft_consensus.cc:2726] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 15:14:11.548219 5167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:11.547988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:44069)
0923 15:14:11.548128 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:11.548169 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260923 15:14:11.582991 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:11.582785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:37217)
0923 15:14:11.582908 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:11.582948 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260923 15:14:11.730983 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:11.730724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:36161)
0923 15:14:11.730872 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:11.730920 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260923 15:14:12.548875 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:12.548666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:55425)
0923 15:14:12.548790 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:12.548827 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260923 15:14:12.583657 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:12.583428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:59405)
0923 15:14:12.583564 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:12.583602 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260923 15:14:12.731665 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:12.731460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:55955)
0923 15:14:12.731582 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:12.731618 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260923 15:14:13.549527 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:13.549327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:34573)
0923 15:14:13.549446 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:13.549481 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260923 15:14:13.584328 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:13.584119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:57275)
0923 15:14:13.584256 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:13.584290 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260923 15:14:13.732414 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:13.732187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:52343)
0923 15:14:13.732329 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:13.732366 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260923 15:14:14.550362 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:14.550120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:56139)
0923 15:14:14.550259 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:14.550301 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260923 15:14:14.585000 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:14.584774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:36287)
0923 15:14:14.584920 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:14.584957 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260923 15:14:14.733086 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:14.732856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:49569)
0923 15:14:14.733000 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:14.733036 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260923 15:14:15.522704 5179 raft_consensus.cc:482] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:15.522825 5179 raft_consensus.cc:504] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4651b86ef91410498814075e11c3595" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 36281 } } peers { permanent_uuid: "bf92d8c74f4d4b4194de19b158da0316" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 46227 } } peers { permanent_uuid: "daa7cdceafab4fd6904356d00bdd53d7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 44879 } }
I20260923 15:14:15.523013 5179 leader_election.cc:288] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4651b86ef91410498814075e11c3595 (127.21.204.67:36281), bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227)
W20260923 15:14:15.523460 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4651b86ef91410498814075e11c3595 (127.21.204.67:36281): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
W20260923 15:14:15.523525 4750 leader_election.cc:341] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf92d8c74f4d4b4194de19b158da0316 (127.21.204.66:46227): Not found: Tablet not found: 6698010813d64dc8baf14832b7cdd68d
I20260923 15:14:15.523541 4750 leader_election.cc:302] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daa7cdceafab4fd6904356d00bdd53d7; no voters: a4651b86ef91410498814075e11c3595, bf92d8c74f4d4b4194de19b158da0316
I20260923 15:14:15.523617 5179 raft_consensus.cc:2726] T 6698010813d64dc8baf14832b7cdd68d P daa7cdceafab4fd6904356d00bdd53d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 15:14:15.551136 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:15.550926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:51837)
0923 15:14:15.551059 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:15.551088 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260923 15:14:15.585632 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:15.585410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:37387)
0923 15:14:15.585554 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:15.585593 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
I20260923 15:14:15.733847 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:15.733611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:37871)
0923 15:14:15.733765 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:15.733800 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260923 15:14:16.551823 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:16.551595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:41535)
0923 15:14:16.551719 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:16.551775 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260923 15:14:16.586349 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:16.586102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:43295)
0923 15:14:16.586248 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:16.586285 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260923 15:14:16.734519 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:16.734304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:56117)
0923 15:14:16.734440 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:16.734475 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260923 15:14:17.552654 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:17.552444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.65:47689)
0923 15:14:17.552574 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:17.552606 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260923 15:14:17.587000 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:17.586816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.66:36089)
0923 15:14:17.586931 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:17.586959 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260923 15:14:17.728021 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 4704
I20260923 15:14:17.733261 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 4866
I20260923 15:14:17.735307 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:17.735061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:41409 (local address 127.21.204.67:40665)
0923 15:14:17.735195 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:17.735239 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260923 15:14:17.738171 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5001
2026-09-23T15:14:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18512 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:14:17 dist-test-slave-t5lt krb5kdc[5191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:14:17 dist-test-slave-t5lt krb5kdc[5191](info): set up 2 sockets
Sep 23 15:14:17 dist-test-slave-t5lt krb5kdc[5191](info): commencing operation
krb5kdc: starting...
W20260923 15:14:18.771284 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:14:18 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:14:18.890753 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43839
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:43839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:18.974054 5204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:18.974262 5204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:18.974287 5204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:18.975519 5204 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 15:14:18.975562 5204 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 15:14:18.975575 5204 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 15:14:18.975587 5204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:14:18.975600 5204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:18.975613 5204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:18.975653 5204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:14:18.975680 5204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:14:18.977185 5204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:43839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:18.977438 5204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:18.977681 5204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:18.983232 5210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:18.983248 5209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:18.983263 5204 server_base.cc:1034] running on GCE node
W20260923 15:14:18.983243 5212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:18.983707 5204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:18.984006 5204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:18.985208 5204 hybrid_clock.cc:648] HybridClock initialized: now 1790176458985184 us; error 38 us; skew 500 ppm
Sep 23 15:14:18 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176458, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:18.987715 5204 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:14:18.988130 5204 webserver.cc:460] Webserver started at http://127.21.204.126:37953/ using document root <none> and password file <none>
I20260923 15:14:18.988386 5204 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:18.988432 5204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:18.988591 5204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:18.989636 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "8db893b01b4641eebfd92236e08e1a7c"
format_stamp: "Formatted at 2026-09-23 15:14:18 on dist-test-slave-t5lt"
server_key: "45068a22a6eb207dc76e8a4209ba4c5b"
server_key_iv: "77c37771d546e214d167a86988b6767d"
server_key_version: "encryptionkey@0"
I20260923 15:14:18.990058 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "8db893b01b4641eebfd92236e08e1a7c"
format_stamp: "Formatted at 2026-09-23 15:14:18 on dist-test-slave-t5lt"
server_key: "45068a22a6eb207dc76e8a4209ba4c5b"
server_key_iv: "77c37771d546e214d167a86988b6767d"
server_key_version: "encryptionkey@0"
I20260923 15:14:18.991467 5204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:18.992333 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:18.992543 5204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:18.992619 5204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "8db893b01b4641eebfd92236e08e1a7c"
format_stamp: "Formatted at 2026-09-23 15:14:18 on dist-test-slave-t5lt"
server_key: "45068a22a6eb207dc76e8a4209ba4c5b"
server_key_iv: "77c37771d546e214d167a86988b6767d"
server_key_version: "encryptionkey@0"
I20260923 15:14:18.992719 5204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:19.000933 5204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:19.002902 5204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:19.003048 5204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:19.007102 5204 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:43839
I20260923 15:14:19.007133 5271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:43839 every 8 connection(s)
I20260923 15:14:19.007493 5204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:14:19.008339 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:19.010660 5272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Bootstrap starting.
I20260923 15:14:19.011250 5272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:19.011555 5272 log.cc:824] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:19.012267 5272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: No bootstrap required, opened a new log
I20260923 15:14:19.013958 5272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:19.014109 5272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:19.014132 5272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db893b01b4641eebfd92236e08e1a7c, State: Initialized, Role: FOLLOWER
I20260923 15:14:19.014295 5272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:19.014369 5272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:14:19.014391 5272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:14:19.014462 5272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:19.014990 5272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:19.015076 5272 leader_election.cc:302] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8db893b01b4641eebfd92236e08e1a7c; no voters:
I20260923 15:14:19.015273 5272 leader_election.cc:288] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:14:19.015316 5275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:14:19.015479 5275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 LEADER]: Becoming Leader. State: Replica: 8db893b01b4641eebfd92236e08e1a7c, State: Running, Role: LEADER
I20260923 15:14:19.015570 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5204
I20260923 15:14:19.015705 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:14:19.015918 22321 external_mini_cluster.cc:1428] Setting key 6f2ca0088cc10a57ed44a06823906671
I20260923 15:14:19.015584 5275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:19.016192 5272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:14:19.016233 5277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db893b01b4641eebfd92236e08e1a7c. Latest consensus state: current_term: 1 leader_uuid: "8db893b01b4641eebfd92236e08e1a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } } }
I20260923 15:14:19.016307 5277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:19.016528 5276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8db893b01b4641eebfd92236e08e1a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } } }
I20260923 15:14:19.016916 5280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:14:19.016981 5276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:19.017539 5280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:14:19.019627 5280 catalog_manager.cc:1347] Generated new cluster ID: 918431f34c9f492a8ef90b9cdf27e344
I20260923 15:14:19.019682 5280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:14:19 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:19.027940 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.016546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.21.204.126:43839)
0923 15:14:19.016704 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.016714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.016953 (+ 239us) server_negotiation.cc:407] Connection header received
0923 15:14:19.017181 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.017189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.017259 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.017377 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:19.018291 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.019473 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.020553 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.021184 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.021312 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.024056 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:19.024060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:19.024061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:19.024272 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:19.026254 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.026650 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.026652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.026653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.026757 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.026966 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.026966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.026967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.027243 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:19.027379 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.027494 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.027705 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260923 15:14:19.033469 5280 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:14:19.034132 5280 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:14:19.044426 5280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Generated new TSK 0
I20260923 15:14:19.044602 5280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:14:19.077168 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43839
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:19.158772 5299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:19.158998 5299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:19.159036 5299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:19.160166 5299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:19.160228 5299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:19.160272 5299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:14:19.161846 5299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:19.162119 5299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:19.162353 5299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:19.167363 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:19.167363 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:19.167402 5299 server_base.cc:1034] running on GCE node
W20260923 15:14:19.167382 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:19.167832 5299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:19.168099 5299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:19.169281 5299 hybrid_clock.cc:648] HybridClock initialized: now 1790176459169260 us; error 31 us; skew 500 ppm
Sep 23 15:14:19 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176459, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:19.171831 5299 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:14:19.172240 5299 webserver.cc:460] Webserver started at http://127.21.204.65:43939/ using document root <none> and password file <none>
I20260923 15:14:19.172475 5299 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:19.172551 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:19.172710 5299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:19.173708 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "278b140ae0364a2784e69a176dfaca7a"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "6667a4b96d814e34225c0b9426a017e6"
server_key_iv: "d4cd52c9be44f402aeb600dbb913a0bc"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.174089 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "278b140ae0364a2784e69a176dfaca7a"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "6667a4b96d814e34225c0b9426a017e6"
server_key_iv: "d4cd52c9be44f402aeb600dbb913a0bc"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.175420 5299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:19.176240 5314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.176446 5299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:19.176570 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "278b140ae0364a2784e69a176dfaca7a"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "6667a4b96d814e34225c0b9426a017e6"
server_key_iv: "d4cd52c9be44f402aeb600dbb913a0bc"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.176661 5299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:19.187659 5299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:19.189484 5299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:19.189646 5299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:19.189970 5299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:19.190280 5299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:19.190335 5299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.190375 5299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:19.190402 5299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.196894 5299 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:36035
I20260923 15:14:19.196935 5427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:36035 every 8 connection(s)
I20260923 15:14:19.197278 5299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:14:19.202481 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5299
I20260923 15:14:19.202613 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:14:19.202821 22321 external_mini_cluster.cc:1428] Setting key 4c4d8e9347ab641e087621be0c8a3dcc
Sep 23 15:14:19 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176459, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:19.207556 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.198128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:42613 (local address 127.21.204.126:43839)
0923 15:14:19.198244 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.198246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.198411 (+ 165us) server_negotiation.cc:407] Connection header received
0923 15:14:19.198957 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.198959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.198972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.199024 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:19.200001 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.200529 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.201534 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.201764 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.201850 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.204349 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:19.204351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:19.204352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:19.204360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:19.206146 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.206563 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.206564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.206565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.206624 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.206883 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.206884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.206884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.207008 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:19.207087 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.207371 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.207463 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 15:14:19.207737 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.198079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:42613)
0923 15:14:19.198278 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.198330 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.198842 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.199091 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.199094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.199699 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:19.199896 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.199901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.200628 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.200630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.201439 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.201441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.201864 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.201864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.202016 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.202635 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:19.202647 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:19.204252 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:19.206249 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:19.206257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:19.206259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:19.206485 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:19.206701 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:19.206702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:19.206702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:19.206816 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:19.207077 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:19.207084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:19.207233 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.207336 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.207497 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260923 15:14:19.208467 5428 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43839
I20260923 15:14:19.208930 5428 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:19.209631 5428 heartbeater.cc:511] Master 127.21.204.126:43839 requested a full tablet report, sending...
I20260923 15:14:19.210645 5236 ts_manager.cc:194] Registered new tserver with Master: 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:14:19.211359 5236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:42613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:14:19.230708 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43839
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:19.311808 5433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:19.312003 5433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:19.312026 5433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:19.313162 5433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:19.313203 5433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:19.313230 5433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:14:19.314699 5433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:19.314930 5433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:19.315177 5433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:19.320443 5441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:19.320510 5439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:19.320492 5438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:19.320570 5433 server_base.cc:1034] running on GCE node
I20260923 15:14:19.320748 5433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:19.320986 5433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:19.322124 5433 hybrid_clock.cc:648] HybridClock initialized: now 1790176459322107 us; error 41 us; skew 500 ppm
Sep 23 15:14:19 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176459, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:19.324504 5433 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:14:19.324898 5433 webserver.cc:460] Webserver started at http://127.21.204.66:41339/ using document root <none> and password file <none>
I20260923 15:14:19.325122 5433 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:19.325165 5433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:19.325256 5433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:19.326324 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "c5d76a86be1a4265a759d84faa293651"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "dd95f8ff529e3c8ea662ddd2dce1b204"
server_key_iv: "054bb47182645de05c68841495615110"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.326627 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "c5d76a86be1a4265a759d84faa293651"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "dd95f8ff529e3c8ea662ddd2dce1b204"
server_key_iv: "054bb47182645de05c68841495615110"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.327971 5433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:14:19.328735 5448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.328912 5433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:19.328994 5433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "c5d76a86be1a4265a759d84faa293651"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "dd95f8ff529e3c8ea662ddd2dce1b204"
server_key_iv: "054bb47182645de05c68841495615110"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.329051 5433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:19.352380 5433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:19.354498 5433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:19.354628 5433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:19.354823 5433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:19.355099 5433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:19.355131 5433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.355154 5433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:19.355167 5433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.360792 5433 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:39273
I20260923 15:14:19.360877 5561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:39273 every 8 connection(s)
I20260923 15:14:19.361181 5433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:14:19.365686 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5433
I20260923 15:14:19.365810 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:14:19.366061 22321 external_mini_cluster.cc:1428] Setting key f7bfd2d578b416a48c48f7f8f6cb982e
Sep 23 15:14:19 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176459, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:19.370585 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.361897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:50487 (local address 127.21.204.126:43839)
0923 15:14:19.361965 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.361967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.362155 (+ 188us) server_negotiation.cc:407] Connection header received
0923 15:14:19.362696 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.362697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.362707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.362763 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:19.363704 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.364193 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.365169 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.365348 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.365396 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.367751 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:19.367753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:19.367753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:19.367761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:19.369154 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.369569 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.369570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.369571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.369632 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.369909 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.369910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.369910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.370034 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:19.370110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.370395 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.370481 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:14:19.370785 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.361820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:50487)
0923 15:14:19.361996 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.362038 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.362604 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.362840 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.362843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.363432 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:19.363621 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.363626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.364296 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.364297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.365079 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.365082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.365421 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.365421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.365539 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.366099 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:19.366110 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:19.367651 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:19.369257 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:19.369265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:19.369267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:19.369488 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:19.369701 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:19.369702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:19.369703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:19.369822 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:19.370116 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:19.370123 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:19.370260 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.370359 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.370518 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260923 15:14:19.371255 5562 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43839
I20260923 15:14:19.371344 5562 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:19.371547 5562 heartbeater.cc:511] Master 127.21.204.126:43839 requested a full tablet report, sending...
I20260923 15:14:19.371925 5236 ts_manager.cc:194] Registered new tserver with Master: c5d76a86be1a4265a759d84faa293651 (127.21.204.66:39273)
I20260923 15:14:19.372327 5236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:50487
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:14:19.393664 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43839
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:19.473802 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:19.474063 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:19.474088 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:19.475189 5567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:19.475224 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:19.475250 5567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:14:19.476630 5567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:19.476805 5567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:19.476966 5567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:19.482086 5572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:19.482080 5575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:19.482123 5567 server_base.cc:1034] running on GCE node
W20260923 15:14:19.482061 5573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:19.482478 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:19.482744 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:19.483929 5567 hybrid_clock.cc:648] HybridClock initialized: now 1790176459483911 us; error 25 us; skew 500 ppm
Sep 23 15:14:19 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176459, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:19.486598 5567 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:14:19.487035 5567 webserver.cc:460] Webserver started at http://127.21.204.67:34453/ using document root <none> and password file <none>
I20260923 15:14:19.487279 5567 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:19.487349 5567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:19.487491 5567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:14:19.488507 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "a39f8c0ebe7744cea8b85e9585cb8630"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "150b0e20ed8afe74f73d5df359f62c49"
server_key_iv: "bfe0f4872afd90511b79eea024e9d281"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.488850 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "a39f8c0ebe7744cea8b85e9585cb8630"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "150b0e20ed8afe74f73d5df359f62c49"
server_key_iv: "bfe0f4872afd90511b79eea024e9d281"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.490226 5567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:14:19.491011 5582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.491243 5567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:19.491354 5567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "a39f8c0ebe7744cea8b85e9585cb8630"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "150b0e20ed8afe74f73d5df359f62c49"
server_key_iv: "bfe0f4872afd90511b79eea024e9d281"
server_key_version: "encryptionkey@0"
I20260923 15:14:19.491441 5567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:19.511461 5567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:19.513489 5567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:19.513651 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:19.513921 5567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:19.514235 5567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:19.514288 5567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.514330 5567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:19.514357 5567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:19.520435 5567 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:41957
I20260923 15:14:19.520481 5695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:41957 every 8 connection(s)
I20260923 15:14:19.520792 5567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:14:19 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176459, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:19.528751 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5567
I20260923 15:14:19.528856 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:14:19.529095 22321 external_mini_cluster.cc:1428] Setting key 3f21240ac7a0d45edd1777d973dc0663
I20260923 15:14:19.530107 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.521453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:57169 (local address 127.21.204.126:43839)
0923 15:14:19.521518 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.521520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.521746 (+ 226us) server_negotiation.cc:407] Connection header received
0923 15:14:19.522330 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.522331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.522343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.522406 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:19.523329 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.523835 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.524790 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.524981 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.525046 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.527204 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:19.527206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:19.527207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:19.527214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:19.528562 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.529063 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.529064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.529064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.529117 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.529431 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.529432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.529432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.529552 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:19.529630 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.529933 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.530013 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:14:19.530303 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.521445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:57169)
0923 15:14:19.521598 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.521648 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.522225 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.522481 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.522484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.523041 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:19.523231 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.523235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.523916 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.523916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.524693 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.524695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.525062 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.525063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.525163 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.525628 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:19.525635 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:19.527117 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:19.528653 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:19.528661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:19.528664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:19.528988 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:19.529234 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:19.529235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:19.529236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:19.529358 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:19.529622 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:19.529629 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:19.529781 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.529915 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.530076 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260923 15:14:19.530712 5696 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43839
I20260923 15:14:19.530789 5696 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:19.530978 5696 heartbeater.cc:511] Master 127.21.204.126:43839 requested a full tablet report, sending...
I20260923 15:14:19.531317 5236 ts_manager.cc:194] Registered new tserver with Master: a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957)
I20260923 15:14:19.531754 5236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:57169
I20260923 15:14:19.540874 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:14:19.548731 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.542409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.21.204.126:43839)
0923 15:14:19.542494 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.542496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.542596 (+ 100us) server_negotiation.cc:407] Connection header received
0923 15:14:19.542703 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.542703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.542715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.542801 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:19.543369 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.543858 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.544619 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.544819 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.544910 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.546126 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:19.546127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:19.546128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:19.546135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:19.547461 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.547846 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.547847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.547848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.547911 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:19.548124 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:19.548125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:19.548125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:19.548297 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:19.548404 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.548548 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.548632 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 15:14:19.550459 5236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:14:19.552115 5236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:14:19.558539 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.554562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.21.204.67:41957)
0923 15:14:19.554953 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.554956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.554965 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:14:19.554997 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.554999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.555011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.555107 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:19.555293 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.556143 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.557335 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.558117 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.558151 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.558194 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.558416 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.558456 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":98,"threads_started":1}
I20260923 15:14:19.558529 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.554414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:41957 (local address 127.0.0.1:46258)
0923 15:14:19.554523 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.554562 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.554646 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.555113 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.555117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.555120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:19.555193 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.555196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.556240 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.556241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.557233 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.557236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.558241 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.558242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.558306 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.558315 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.558399 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.558454 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":66,"threads_started":1}
I20260923 15:14:19.559109 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.554443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:39273 (local address 127.0.0.1:34836)
0923 15:14:19.554980 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.554990 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.555045 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.555386 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.555386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.555389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:19.555438 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.555440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.556824 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.556825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.557916 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.557920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.558855 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.558856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.558925 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.558929 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.558989 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.559025 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260923 15:14:19.559163 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.554775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.21.204.66:39273)
0923 15:14:19.555049 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.555053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.555079 (+ 26us) server_negotiation.cc:407] Connection header received
0923 15:14:19.555119 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.555121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.555135 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.555303 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:19.555519 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.556707 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.557995 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.558695 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.558763 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.558817 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.559026 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.559073 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260923 15:14:19.559232 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.554763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.21.204.65:36035)
0923 15:14:19.554966 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.554972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.555334 (+ 362us) server_negotiation.cc:407] Connection header received
0923 15:14:19.555378 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.555380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.555395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.555461 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:19.555670 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.556751 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.557932 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.558690 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.558763 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.558816 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.559071 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.559116 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260923 15:14:19.559581 5630 tablet_service.cc:1467] Processing CreateTablet for tablet a34700fe30a04ba1aaca981ddd29d472 (DEFAULT_TABLE table=test-table [id=e55dcec373be48f39da676f9b04ce5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:19.559662 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.554634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36035 (local address 127.0.0.1:58176)
0923 15:14:19.555215 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.555226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.555274 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.555532 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.555533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.555535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:19.555580 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.555582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.556859 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.556860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.557824 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.557826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.558855 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.558856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.558925 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.558930 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.558980 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.559026 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":5,"thread_start_us":52,"threads_started":1}
I20260923 15:14:19.559973 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34700fe30a04ba1aaca981ddd29d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:19.560359 5496 tablet_service.cc:1467] Processing CreateTablet for tablet a34700fe30a04ba1aaca981ddd29d472 (DEFAULT_TABLE table=test-table [id=e55dcec373be48f39da676f9b04ce5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:19.560360 5362 tablet_service.cc:1467] Processing CreateTablet for tablet a34700fe30a04ba1aaca981ddd29d472 (DEFAULT_TABLE table=test-table [id=e55dcec373be48f39da676f9b04ce5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:19.560670 5496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34700fe30a04ba1aaca981ddd29d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:19.560669 5362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34700fe30a04ba1aaca981ddd29d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:19.562660 5713 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Bootstrap starting.
I20260923 15:14:19.563278 5713 tablet_bootstrap.cc:654] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:19.563789 5713 log.cc:824] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:19.563953 5714 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap starting.
I20260923 15:14:19.564539 5715 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Bootstrap starting.
I20260923 15:14:19.564625 5714 tablet_bootstrap.cc:654] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:19.564790 5713 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: No bootstrap required, opened a new log
I20260923 15:14:19.564874 5713 ts_tablet_manager.cc:1397] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:14:19.565188 5714 log.cc:824] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:19.565227 5715 tablet_bootstrap.cc:654] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:19.565773 5715 log.cc:824] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:19.566030 5714 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: No bootstrap required, opened a new log
I20260923 15:14:19.566124 5714 ts_tablet_manager.cc:1397] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:14:19.566548 5715 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: No bootstrap required, opened a new log
I20260923 15:14:19.566622 5715 ts_tablet_manager.cc:1397] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:14:19.566825 5713 raft_consensus.cc:348] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.566987 5713 raft_consensus.cc:374] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:19.567032 5713 raft_consensus.cc:729] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f8c0ebe7744cea8b85e9585cb8630, State: Initialized, Role: FOLLOWER
I20260923 15:14:19.567139 5713 consensus_queue.cc:260] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.567698 5713 ts_tablet_manager.cc:1428] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:14:19.567734 5696 heartbeater.cc:503] Master 127.21.204.126:43839 was elected leader, sending a full tablet report...
I20260923 15:14:19.568516 5714 raft_consensus.cc:348] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.568698 5714 raft_consensus.cc:374] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:19.568605 5715 raft_consensus.cc:348] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.568743 5715 raft_consensus.cc:374] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:19.568742 5714 raft_consensus.cc:729] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278b140ae0364a2784e69a176dfaca7a, State: Initialized, Role: FOLLOWER
I20260923 15:14:19.568810 5715 raft_consensus.cc:729] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d76a86be1a4265a759d84faa293651, State: Initialized, Role: FOLLOWER
I20260923 15:14:19.568894 5714 consensus_queue.cc:260] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.569041 5715 consensus_queue.cc:260] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.569192 5428 heartbeater.cc:503] Master 127.21.204.126:43839 was elected leader, sending a full tablet report...
I20260923 15:14:19.569177 5714 ts_tablet_manager.cc:1428] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:14:19.569258 5720 raft_consensus.cc:482] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:19.569325 5720 raft_consensus.cc:504] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.569416 5715 ts_tablet_manager.cc:1428] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:14:19.569427 5562 heartbeater.cc:503] Master 127.21.204.126:43839 was elected leader, sending a full tablet report...
I20260923 15:14:19.570165 5720 leader_election.cc:288] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957), 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
I20260923 15:14:19.572516 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.569751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:41957 (local address 127.21.204.66:56491)
0923 15:14:19.569813 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.569823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.569861 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.570164 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.570166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.570168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:19.570244 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.570246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.570961 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.570962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.571692 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.571694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.572245 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.572245 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.572323 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.572328 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.572408 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.572456 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 15:14:19.572513 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.569823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:56491 (local address 127.21.204.67:41957)
0923 15:14:19.569865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.569867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.569927 (+ 60us) server_negotiation.cc:407] Connection header received
0923 15:14:19.569949 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.569950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.569960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.570007 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:19.570330 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.570874 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.571784 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.572143 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.572244 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.572253 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.572416 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.572451 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:14:19.572800 5650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34700fe30a04ba1aaca981ddd29d472" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" is_pre_election: true
I20260923 15:14:19.572906 5650 raft_consensus.cc:2454] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 0.
I20260923 15:14:19.573042 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.570093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:44631 (local address 127.21.204.65:36035)
0923 15:14:19.570130 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.570132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.570428 (+ 296us) server_negotiation.cc:407] Connection header received
0923 15:14:19.570522 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.570523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.570533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.570581 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:19.570728 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.571171 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.572225 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.572670 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.572736 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.572743 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.572933 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.572970 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:14:19.573130 5451 leader_election.cc:302] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a39f8c0ebe7744cea8b85e9585cb8630, c5d76a86be1a4265a759d84faa293651; no voters:
I20260923 15:14:19.573256 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.570051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36035 (local address 127.21.204.66:44631)
0923 15:14:19.570370 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.570380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:19.570456 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:19.570582 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:19.570582 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:19.570584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:19.570650 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.570652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.571251 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.571252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.572141 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:19.572144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.572776 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.572777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:19.572846 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.572850 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:14:19.573122 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 15:14:19.573175 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":75,"threads_started":1}
I20260923 15:14:19.573259 5722 raft_consensus.cc:2781] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:14:19.573338 5722 raft_consensus.cc:482] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:14:19.573374 5722 raft_consensus.cc:3037] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:19.573617 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34700fe30a04ba1aaca981ddd29d472" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278b140ae0364a2784e69a176dfaca7a" is_pre_election: true
I20260923 15:14:19.573738 5382 raft_consensus.cc:2454] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 0.
I20260923 15:14:19.574205 5722 raft_consensus.cc:504] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.574396 5722 leader_election.cc:288] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 election: Requested vote from peers a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957), 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
I20260923 15:14:19.574466 5650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34700fe30a04ba1aaca981ddd29d472" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39f8c0ebe7744cea8b85e9585cb8630"
I20260923 15:14:19.574486 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34700fe30a04ba1aaca981ddd29d472" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278b140ae0364a2784e69a176dfaca7a"
I20260923 15:14:19.574532 5650 raft_consensus.cc:3037] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:19.574550 5382 raft_consensus.cc:3037] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:19.575109 5382 raft_consensus.cc:2454] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 1.
I20260923 15:14:19.575147 5650 raft_consensus.cc:2454] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 1.
I20260923 15:14:19.575305 5451 leader_election.cc:302] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a39f8c0ebe7744cea8b85e9585cb8630, c5d76a86be1a4265a759d84faa293651; no voters:
I20260923 15:14:19.575402 5722 raft_consensus.cc:2781] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:14:19.575456 5722 raft_consensus.cc:686] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 1 LEADER]: Becoming Leader. State: Replica: c5d76a86be1a4265a759d84faa293651, State: Running, Role: LEADER
I20260923 15:14:19.575533 5722 consensus_queue.cc:237] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:19.576261 5236 catalog_manager.cc:5495] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5d76a86be1a4265a759d84faa293651 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "c5d76a86be1a4265a759d84faa293651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } health_report { overall_health: UNKNOWN } } }
I20260923 15:14:19.587495 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.585128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.21.204.66:39273)
0923 15:14:19.585207 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:19.585209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:19.585218 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:14:19.585256 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:19.585257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:19.585274 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:19.585351 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:14:19.585489 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.586007 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.586749 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:19.586935 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:19.587018 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:19.587197 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:14:19.587289 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:14:19.587381 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:19.587400 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 15:14:19.587419 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 15:14:19.589475 5382 raft_consensus.cc:1261] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Refusing update from remote peer c5d76a86be1a4265a759d84faa293651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:19.589545 5650 raft_consensus.cc:1261] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Refusing update from remote peer c5d76a86be1a4265a759d84faa293651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:19.589872 5722 consensus_queue.cc:1035] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:19.589999 5720 consensus_queue.cc:1035] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:19.592166 5726 mvcc.cc:205] Tried to move back new op lower bound from 7332562778474876928 to 7332562778421989376. Current Snapshot: MvccSnapshot[applied={T|T < 7332562778474876928}]
I20260923 15:14:19.592548 5727 mvcc.cc:205] Tried to move back new op lower bound from 7332562778474876928 to 7332562778421989376. Current Snapshot: MvccSnapshot[applied={T|T < 7332562778474876928}]
I20260923 15:14:19.592886 5728 mvcc.cc:205] Tried to move back new op lower bound from 7332562778474876928 to 7332562778421989376. Current Snapshot: MvccSnapshot[applied={T|T < 7332562778474876928}]
I20260923 15:14:19.594621 5235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:19.594730 5235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:19.595489 5235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Sending DeleteTablet for 3 replicas of tablet a34700fe30a04ba1aaca981ddd29d472
I20260923 15:14:19.595808 5496 tablet_service.cc:1514] Processing DeleteTablet for tablet a34700fe30a04ba1aaca981ddd29d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:19 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:34836
I20260923 15:14:19.595852 5362 tablet_service.cc:1514] Processing DeleteTablet for tablet a34700fe30a04ba1aaca981ddd29d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:19 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:58176
I20260923 15:14:19.595853 5630 tablet_service.cc:1514] Processing DeleteTablet for tablet a34700fe30a04ba1aaca981ddd29d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:19 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:46258
I20260923 15:14:19.596072 5737 tablet_replica.cc:330] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: stopping tablet replica
I20260923 15:14:19.596113 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5299
I20260923 15:14:19.596197 5735 tablet_replica.cc:330] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: stopping tablet replica
I20260923 15:14:19.596215 5737 raft_consensus.cc:2229] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:19.596339 5737 raft_consensus.cc:2258] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:19.596933 5735 raft_consensus.cc:2229] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:14:19.597024 5735 raft_consensus.cc:2258] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:19.597039 5737 ts_tablet_manager.cc:1905] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:19.597438 5735 ts_tablet_manager.cc:1905] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:19.598722 5737 ts_tablet_manager.cc:1918] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:19.598785 5737 log.cc:1196] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/a34700fe30a04ba1aaca981ddd29d472
I20260923 15:14:19.599071 5737 ts_tablet_manager.cc:1939] T a34700fe30a04ba1aaca981ddd29d472 P a39f8c0ebe7744cea8b85e9585cb8630: Deleting consensus metadata
I20260923 15:14:19.599179 5735 ts_tablet_manager.cc:1918] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:19.599352 5735 log.cc:1196] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/a34700fe30a04ba1aaca981ddd29d472
I20260923 15:14:19.599668 5735 ts_tablet_manager.cc:1939] T a34700fe30a04ba1aaca981ddd29d472 P c5d76a86be1a4265a759d84faa293651: Deleting consensus metadata
I20260923 15:14:19.599805 5222 catalog_manager.cc:4841] TS a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957): tablet a34700fe30a04ba1aaca981ddd29d472 (table test-table [id=e55dcec373be48f39da676f9b04ce5ee]) successfully deleted
I20260923 15:14:19.600075 5223 catalog_manager.cc:4841] TS c5d76a86be1a4265a759d84faa293651 (127.21.204.66:39273): tablet a34700fe30a04ba1aaca981ddd29d472 (table test-table [id=e55dcec373be48f39da676f9b04ce5ee]) successfully deleted
W20260923 15:14:19.603415 5223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:36035 (error 108)
I20260923 15:14:19.603652 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5433
I20260923 15:14:19.606287 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:19.606155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36035 (local address 127.0.0.1:58180)
0923 15:14:19.606210 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:19.606249 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:36035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 15:14:19.606403 5223 catalog_manager.cc:4584] TS 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a34700fe30a04ba1aaca981ddd29d472: Network error: Client connection negotiation failed: client connection to 127.21.204.65:36035: connect: Connection refused (error 111)
I20260923 15:14:19.609099 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5567
I20260923 15:14:19.613857 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5204
I20260923 15:14:39.619383 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43839
--webserver_interface=127.21.204.126
--webserver_port=37953
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:43839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:14:39.699942 5739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:39.700181 5739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:39.700218 5739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:39.701401 5739 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 15:14:39.701467 5739 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 15:14:39.701491 5739 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 15:14:39.701521 5739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:14:39.701556 5739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:39.701586 5739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:39.701610 5739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:14:39.701635 5739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:14:39.703182 5739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:43839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=37953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:39.703492 5739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:39.703722 5739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:39.712746 5748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:39.712728 5746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:39.712729 5739 server_base.cc:1034] running on GCE node
W20260923 15:14:39.712796 5745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:39.713208 5739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:39.713481 5739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:39.714622 5739 hybrid_clock.cc:648] HybridClock initialized: now 1790176479714607 us; error 30 us; skew 500 ppm
Sep 23 15:14:39 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176479, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:39.717039 5739 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:14:39.717505 5739 webserver.cc:460] Webserver started at http://127.21.204.126:37953/ using document root <none> and password file <none>
I20260923 15:14:39.717702 5739 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:39.717756 5739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:39.719185 5739 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:39.719888 5755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:39.720090 5739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:39.720183 5739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "8db893b01b4641eebfd92236e08e1a7c"
format_stamp: "Formatted at 2026-09-23 15:14:18 on dist-test-slave-t5lt"
server_key: "45068a22a6eb207dc76e8a4209ba4c5b"
server_key_iv: "77c37771d546e214d167a86988b6767d"
server_key_version: "encryptionkey@0"
I20260923 15:14:39.720578 5739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:39.734267 5739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:39.736387 5739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:39.736505 5739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:39.740615 5739 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:43839
I20260923 15:14:39.740669 5807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:43839 every 8 connection(s)
I20260923 15:14:39.741159 5739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:14:39.742856 5808 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:14:39.743484 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Bootstrap starting.
I20260923 15:14:39.744467 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5739
I20260923 15:14:39.744802 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:36035
--local_ip_for_outbound_sockets=127.21.204.65
--tserver_master_addrs=127.21.204.126:43839
--webserver_port=43939
--webserver_interface=127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:14:39.752720 5808 log.cc:824] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:39.754746 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:14:39.755059 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Bootstrap complete.
I20260923 15:14:39.757345 5808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:39.757571 5808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db893b01b4641eebfd92236e08e1a7c, State: Initialized, Role: FOLLOWER
I20260923 15:14:39.757753 5808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:39.757812 5808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:14:39.757898 5808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:14:39.757942 5808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 1 FOLLOWER]: Advancing to term 2
I20260923 15:14:39.759078 5808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:39.759176 5808 leader_election.cc:302] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8db893b01b4641eebfd92236e08e1a7c; no voters:
I20260923 15:14:39.759413 5808 leader_election.cc:288] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 15:14:39.759449 5812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 FOLLOWER]: Leader election won for term 2
I20260923 15:14:39.759570 5812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 LEADER]: Becoming Leader. State: Replica: 8db893b01b4641eebfd92236e08e1a7c, State: Running, Role: LEADER
I20260923 15:14:39.759689 5808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:14:39.759730 5812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:14:39.760100 5814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8db893b01b4641eebfd92236e08e1a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } } }
I20260923 15:14:39.760102 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db893b01b4641eebfd92236e08e1a7c. Latest consensus state: current_term: 2 leader_uuid: "8db893b01b4641eebfd92236e08e1a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } } }
I20260923 15:14:39.760210 5814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:39.760385 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: This master's current role is: LEADER
I20260923 15:14:39.760586 5821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:14:39.761529 5821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:14:39.761716 5821 catalog_manager.cc:1259] Loaded cluster ID: 918431f34c9f492a8ef90b9cdf27e344
I20260923 15:14:39.761936 5821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:14:39.762753 5821 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:14:39.762943 5821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Loaded TSK: 0
I20260923 15:14:39.771749 5821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Generated new TSK 1
I20260923 15:14:39.771855 5821 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 15:14:39.839668 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:39.839906 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:39.839946 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:39.841079 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:39.841141 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:39.841187 5810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:14:39.842854 5810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:36035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=43939
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:39.843106 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:39.843458 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:39.848351 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:39.848443 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:39.848344 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:39.848762 5810 server_base.cc:1034] running on GCE node
I20260923 15:14:39.848990 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:39.849241 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:39.850402 5810 hybrid_clock.cc:648] HybridClock initialized: now 1790176479850387 us; error 31 us; skew 500 ppm
Sep 23 15:14:39 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176479, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:39.852950 5810 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:14:39.853443 5810 webserver.cc:460] Webserver started at http://127.21.204.65:43939/ using document root <none> and password file <none>
I20260923 15:14:39.853674 5810 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:39.853757 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:39.855108 5810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:39.855804 5845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:39.855985 5810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:39.856096 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "278b140ae0364a2784e69a176dfaca7a"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "6667a4b96d814e34225c0b9426a017e6"
server_key_iv: "d4cd52c9be44f402aeb600dbb913a0bc"
server_key_version: "encryptionkey@0"
I20260923 15:14:39.856456 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:39.866721 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:39.869020 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:39.869179 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:39.869460 5810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:39.869963 5852 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 15:14:39.871400 5810 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 15:14:39.871471 5810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 15:14:39.871515 5810 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 15:14:39.872138 5810 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 15:14:39.872185 5810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 15:14:39.872238 5852 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap starting.
I20260923 15:14:39.880041 5810 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:36035
I20260923 15:14:39.880204 5960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:36035 every 8 connection(s)
I20260923 15:14:39.880709 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:14:39.884109 5852 log.cc:824] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:39.888051 5852 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:14:39.888480 5852 tablet_bootstrap.cc:492] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap complete.
I20260923 15:14:39.889765 5852 ts_tablet_manager.cc:1397] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
Sep 23 15:14:39 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176479, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:39.890429 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5810
I20260923 15:14:39.890738 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:39273
--local_ip_for_outbound_sockets=127.21.204.66
--tserver_master_addrs=127.21.204.126:43839
--webserver_port=41339
--webserver_interface=127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:14:39.891531 5852 raft_consensus.cc:348] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:39.891948 5852 raft_consensus.cc:729] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278b140ae0364a2784e69a176dfaca7a, State: Initialized, Role: FOLLOWER
I20260923 15:14:39.892123 5852 consensus_queue.cc:260] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:39.892452 5852 ts_tablet_manager.cc:1428] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 15:14:39.894558 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:39.881444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:46031)
0923 15:14:39.881733 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:39.881777 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:39.882385 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:39.883077 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:39.883082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:39.883544 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:39.883824 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:39.883831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.885549 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.885550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:39.886897 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:39.886900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.887725 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.887726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:39.887878 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:39.888504 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:39.888515 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:39.890451 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:39.892693 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:39.892700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:39.892702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:39.892971 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:39.893335 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:39.893337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:39.893337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:39.893452 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:39.893863 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:39.893871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:39.894042 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 15:14:39.894152 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 15:14:39.894336 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
I20260923 15:14:39.894586 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:39.881561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:46031 (local address 127.21.204.126:43839)
0923 15:14:39.882603 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:39.882612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:39.882647 (+ 35us) server_negotiation.cc:407] Connection header received
0923 15:14:39.882789 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:39.882795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:39.882849 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:39.882985 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:39.883980 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.885395 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.887003 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.887552 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.887746 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:39.890578 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:39.890583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:39.890583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:39.890595 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:39.892584 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:39.893076 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:39.893079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:39.893079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:39.893172 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:39.893541 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:39.893542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:39.893542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:39.893776 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:39.893926 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:39.894226 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 15:14:39.894372 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":99,"threads_started":1}
I20260923 15:14:39.895274 5961 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43839
I20260923 15:14:39.895372 5961 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:39.895535 5961 heartbeater.cc:511] Master 127.21.204.126:43839 requested a full tablet report, sending...
I20260923 15:14:39.896386 5772 ts_manager.cc:194] Registered new tserver with Master: 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
I20260923 15:14:39.897819 5772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:46031
I20260923 15:14:39.898993 5961 heartbeater.cc:503] Master 127.21.204.126:43839 was elected leader, sending a full tablet report...
Sep 23 15:14:39 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176479, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:14:39.907682 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:39.897060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36035 (local address 127.0.0.1:32768)
0923 15:14:39.897420 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:39.897455 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:39.897515 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:39.897812 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:39.897817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:39.898351 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:39.898405 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:39.898407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.899991 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.899993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:39.901138 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:39.901142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.901780 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.901782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:39.901925 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:39.902452 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:39.902459 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:39.904213 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:39.906389 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:39.906396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:39.906398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:39.906661 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:39.906899 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:39.906900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:39.906900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:39.906971 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:39.907288 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:39.907295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:39.907352 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 15:14:39.907445 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 15:14:39.907568 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":66,"threads_started":1}
I20260923 15:14:39.907727 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:39.897274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.21.204.65:36035)
0923 15:14:39.897626 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:39.897630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:39.897639 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:14:39.897668 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:39.897682 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:39.897697 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:39.897754 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:39.898493 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.899886 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.901239 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:39.901668 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:39.901748 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:39.904312 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:39.904321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:39.904322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:39.904331 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:39.906288 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:39.906751 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:39.906753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:39.906754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:39.906819 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:39.907038 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:39.907039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:39.907039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:39.907211 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:39.907293 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:39.907532 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 15:14:39.907631 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":325,"thread_start_us":67,"threads_started":1}
I20260923 15:14:39.908574 5894 tablet_service.cc:1514] Processing DeleteTablet for tablet a34700fe30a04ba1aaca981ddd29d472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:19 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:32768
I20260923 15:14:39.909183 5970 tablet_replica.cc:330] stopping tablet replica
I20260923 15:14:39.909297 5970 raft_consensus.cc:2229] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:39.909370 5970 raft_consensus.cc:2258] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:39.909686 5970 ts_tablet_manager.cc:1905] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:39.912283 5970 ts_tablet_manager.cc:1918] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:39.912388 5970 log.cc:1196] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal/wals/a34700fe30a04ba1aaca981ddd29d472
I20260923 15:14:39.912667 5970 ts_tablet_manager.cc:1939] T a34700fe30a04ba1aaca981ddd29d472 P 278b140ae0364a2784e69a176dfaca7a: Deleting consensus metadata
I20260923 15:14:39.913316 5759 catalog_manager.cc:4841] TS 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035): tablet a34700fe30a04ba1aaca981ddd29d472 (table test-table [id=e55dcec373be48f39da676f9b04ce5ee]) successfully deleted
W20260923 15:14:39.987308 5966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:39.987555 5966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:39.987594 5966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:39.988740 5966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:39.988812 5966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:39.988859 5966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:14:39.990360 5966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:39273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=41339
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:39.990628 5966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:39.990866 5966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:39.995947 5975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:39.995958 5976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:39.995962 5978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:39.996249 5966 server_base.cc:1034] running on GCE node
I20260923 15:14:39.996461 5966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:39.996732 5966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:39.997927 5966 hybrid_clock.cc:648] HybridClock initialized: now 1790176479997910 us; error 27 us; skew 500 ppm
Sep 23 15:14:39 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176479, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:40.000444 5966 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:14:40.000828 5966 webserver.cc:460] Webserver started at http://127.21.204.66:41339/ using document root <none> and password file <none>
I20260923 15:14:40.001046 5966 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:40.001104 5966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:40.002602 5966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:40.003276 5985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.003484 5966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:14:40.003577 5966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "c5d76a86be1a4265a759d84faa293651"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "dd95f8ff529e3c8ea662ddd2dce1b204"
server_key_iv: "054bb47182645de05c68841495615110"
server_key_version: "encryptionkey@0"
I20260923 15:14:40.003927 5966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:40.020448 5966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:40.022485 5966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:40.022595 5966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:40.022789 5966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:40.023113 5966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:40.023146 5966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.023168 5966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:40.023181 5966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.029134 5966 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:39273
I20260923 15:14:40.029193 6100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:39273 every 8 connection(s)
I20260923 15:14:40.029559 5966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:14:40 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176479, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:40.038221 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 5966
I20260923 15:14:40.038542 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:41957
--local_ip_for_outbound_sockets=127.21.204.67
--tserver_master_addrs=127.21.204.126:43839
--webserver_port=34453
--webserver_interface=127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:14:40.039856 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.030371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:56169 (local address 127.21.204.126:43839)
0923 15:14:40.030441 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.030443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.030728 (+ 285us) server_negotiation.cc:407] Connection header received
0923 15:14:40.031389 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.031391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.031422 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.031488 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:40.032539 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.033247 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.034642 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.034855 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.034919 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.036910 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:40.036911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:40.036912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:40.036918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:40.038464 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.038942 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.038943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.038943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.038995 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.039263 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.039264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.039264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.039363 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:40.039414 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.039722 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.039793 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:14:40.040166 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.030354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:56169)
0923 15:14:40.030600 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.030645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:40.031274 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:40.031574 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:40.031578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:40.032178 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:40.032371 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.032376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.033364 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.033365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.034551 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.034554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.034936 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.034937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.035034 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.035461 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:40.035470 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:40.036818 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:40.038538 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:40.038546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:40.038548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:40.038860 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:40.039057 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:40.039058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:40.039058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:40.039200 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:40.039427 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:40.039433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:40.039592 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 15:14:40.039703 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 15:14:40.039897 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20260923 15:14:40.040652 6101 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43839
I20260923 15:14:40.040752 6101 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:40.040916 6101 heartbeater.cc:511] Master 127.21.204.126:43839 requested a full tablet report, sending...
I20260923 15:14:40.041289 5772 ts_manager.cc:194] Registered new tserver with Master: c5d76a86be1a4265a759d84faa293651 (127.21.204.66:39273)
I20260923 15:14:40.041832 5772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:56169
W20260923 15:14:40.119727 6104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:14:40.119920 6104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:14:40.119943 6104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:14:40.121035 6104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:14:40.121073 6104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:14:40.121098 6104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:14:40.122515 6104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:41957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=34453
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:14:40.122787 6104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:14:40.122982 6104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:14:40.127948 6109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:40.127945 6112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:14:40.127946 6110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:14:40.128278 6104 server_base.cc:1034] running on GCE node
I20260923 15:14:40.128476 6104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:14:40.128706 6104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:14:40.129858 6104 hybrid_clock.cc:648] HybridClock initialized: now 1790176480129834 us; error 44 us; skew 500 ppm
Sep 23 15:14:40 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176480, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:14:40.132396 6104 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:14:40.132815 6104 webserver.cc:460] Webserver started at http://127.21.204.67:34453/ using document root <none> and password file <none>
I20260923 15:14:40.133050 6104 fs_manager.cc:359] Metadata directory not provided
I20260923 15:14:40.133105 6104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:14:40.134521 6104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 15:14:40.135105 6119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.135293 6104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.135391 6104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "a39f8c0ebe7744cea8b85e9585cb8630"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "150b0e20ed8afe74f73d5df359f62c49"
server_key_iv: "bfe0f4872afd90511b79eea024e9d281"
server_key_version: "encryptionkey@0"
I20260923 15:14:40.135720 6104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:14:40.165166 6104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:14:40.167271 6104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:14:40.167448 6104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:14:40.167696 6104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:14:40.168018 6104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:14:40.168072 6104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.168112 6104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:14:40.168149 6104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.173965 6104 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:41957
I20260923 15:14:40.174021 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:41957 every 8 connection(s)
I20260923 15:14:40.174374 6104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:14:40 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176480, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:14:40.183843 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.175110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:48567 (local address 127.21.204.126:43839)
0923 15:14:40.175159 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.175160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.175426 (+ 266us) server_negotiation.cc:407] Connection header received
0923 15:14:40.175959 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.175961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.175975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.176027 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:40.176951 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.177441 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.178464 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.178661 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.178758 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.180959 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:40.180961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:40.180961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:40.180970 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:40.182408 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.182853 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.182855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.182855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.182916 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.183193 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.183194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.183194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.183311 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:40.183383 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.183663 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.183753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:14:40.184046 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.175109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:48567)
0923 15:14:40.175306 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.175341 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:40.175851 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:40.176094 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:40.176098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:40.176658 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:14:40.176847 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.176852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.177540 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.177541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.178357 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.178360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.178776 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.178777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.178905 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.179408 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:14:40.179417 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:14:40.180850 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:14:40.182536 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:40.182543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:40.182545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:40.182771 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:40.182985 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:14:40.182985 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:14:40.182986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:14:40.183102 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:14:40.183386 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:14:40.183394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:14:40.183526 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 15:14:40.183633 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 15:14:40.183786 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260923 15:14:40.184549 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 6104
I20260923 15:14:40.184625 6233 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43839
I20260923 15:14:40.184718 6233 heartbeater.cc:464] Registering TS with master...
I20260923 15:14:40.184923 6233 heartbeater.cc:511] Master 127.21.204.126:43839 requested a full tablet report, sending...
I20260923 15:14:40.185374 5772 ts_manager.cc:194] Registered new tserver with Master: a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957)
I20260923 15:14:40.185750 5772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:48567
I20260923 15:14:40.191555 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.185001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.21.204.126:43839)
0923 15:14:40.185058 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.185060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.185302 (+ 242us) server_negotiation.cc:407] Connection header received
0923 15:14:40.185369 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.185370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.185382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.185435 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:40.186216 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.186733 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.187589 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.187816 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.187895 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.189104 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:40.189106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:40.189106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:40.189113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:40.190395 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.190771 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.190772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.190773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.190831 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.191034 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.191035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.191035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.191147 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:40.191345 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.191383 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.191457 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:14:40.192451 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:14:40.195067 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.192852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.21.204.126:43839)
0923 15:14:40.192925 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.192927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.193014 (+ 87us) server_negotiation.cc:407] Connection header received
0923 15:14:40.193065 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.193065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.193099 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.193162 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:14:40.193266 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.193715 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.194483 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.194668 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.194731 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.194840 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:14:40.194879 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 15:14:40.194994 (+ 115us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51058: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260923 15:14:40.195140 5964 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51058: token expired
W20260923 15:14:40.195218 5700 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.204.126:43839)
I20260923 15:14:40.200631 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.195495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.21.204.126:43839)
0923 15:14:40.195530 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.195531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.195539 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:14:40.195601 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.195601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.195609 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.195661 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:14:40.196122 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.196548 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.197253 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.197427 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.197489 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.198588 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:14:40.198589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:14:40.198589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:14:40.198595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:14:40.199733 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.200031 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.200032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.200032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.200081 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:14:40.200257 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:14:40.200258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:14:40.200258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:14:40.200349 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:14:40.200421 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.200503 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.200554 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:14:40.201782 5772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:14:40.203431 5772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:14:40.205621 5894 tablet_service.cc:1467] Processing CreateTablet for tablet cacadb262746463aabc79dbd17b10607 (DEFAULT_TABLE table=test-table [id=53e3c71e8f44440eb5e5369a2af3dc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:40.205855 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacadb262746463aabc79dbd17b10607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:40.207422 5852 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap starting.
I20260923 15:14:40.207754 5852 tablet_bootstrap.cc:654] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:40.208618 5852 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: No bootstrap required, opened a new log
I20260923 15:14:40.208680 5852 ts_tablet_manager.cc:1397] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 15:14:40.208839 5852 raft_consensus.cc:348] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.208913 5852 raft_consensus.cc:374] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:40.208959 5852 raft_consensus.cc:729] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278b140ae0364a2784e69a176dfaca7a, State: Initialized, Role: FOLLOWER
I20260923 15:14:40.209010 5852 consensus_queue.cc:260] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.209100 5852 ts_tablet_manager.cc:1428] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 15:14:40.209854 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.205303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:41957 (local address 127.0.0.1:42996)
0923 15:14:40.205537 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.205553 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:40.205595 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:40.205892 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:40.205894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:40.205897 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:40.205942 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.205943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.207221 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.207222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.208505 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.208508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.209512 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.209513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.209580 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.209657 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 15:14:40.209716 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:14:40.209759 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":70,"threads_started":1}
I20260923 15:14:40.209918 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.205517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.21.204.67:41957)
0923 15:14:40.205704 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.205709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.205719 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:14:40.205760 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.205762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.205779 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.205863 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:40.206008 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.207132 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.208611 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.209437 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.209511 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.209572 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.209741 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.209789 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260923 15:14:40.210024 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.205364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.21.204.66:39273)
0923 15:14:40.205748 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.205752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.205762 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:14:40.205803 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.205805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.205820 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.206070 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:40.206156 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.207343 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.208780 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.209539 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.209640 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.209710 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.209839 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.209924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":91,"threads_started":1}
I20260923 15:14:40.209969 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.205260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:39273 (local address 127.0.0.1:47340)
0923 15:14:40.205311 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.205322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:40.205360 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:40.205941 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:40.205942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:40.205944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:40.205990 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.205991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.207459 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.207461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.208600 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.208603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.209644 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.209645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.209741 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.209746 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:14:40.209810 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 15:14:40.209867 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 15:14:40.210811 6167 tablet_service.cc:1467] Processing CreateTablet for tablet cacadb262746463aabc79dbd17b10607 (DEFAULT_TABLE table=test-table [id=53e3c71e8f44440eb5e5369a2af3dc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:40.210814 6034 tablet_service.cc:1467] Processing CreateTablet for tablet cacadb262746463aabc79dbd17b10607 (DEFAULT_TABLE table=test-table [id=53e3c71e8f44440eb5e5369a2af3dc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:14:40.211138 6034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacadb262746463aabc79dbd17b10607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:40.211139 6167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacadb262746463aabc79dbd17b10607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:14:40.213766 6242 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Bootstrap starting.
I20260923 15:14:40.213944 6243 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Bootstrap starting.
I20260923 15:14:40.214524 6242 tablet_bootstrap.cc:654] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:40.214536 6243 tablet_bootstrap.cc:654] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Neither blocks nor log segments found. Creating new log.
I20260923 15:14:40.215003 6243 log.cc:824] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:40.215030 6242 log.cc:824] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Log is configured to *not* fsync() on all Append() calls
I20260923 15:14:40.215866 6243 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: No bootstrap required, opened a new log
I20260923 15:14:40.215874 6242 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: No bootstrap required, opened a new log
I20260923 15:14:40.215947 6242 ts_tablet_manager.cc:1397] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:14:40.215947 6243 ts_tablet_manager.cc:1397] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:14:40.217947 6242 raft_consensus.cc:348] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.217965 6243 raft_consensus.cc:348] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.218137 6243 raft_consensus.cc:374] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:40.218137 6242 raft_consensus.cc:374] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:14:40.218173 6242 raft_consensus.cc:729] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f8c0ebe7744cea8b85e9585cb8630, State: Initialized, Role: FOLLOWER
I20260923 15:14:40.218173 6243 raft_consensus.cc:729] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d76a86be1a4265a759d84faa293651, State: Initialized, Role: FOLLOWER
I20260923 15:14:40.218300 6243 consensus_queue.cc:260] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.218302 6242 consensus_queue.cc:260] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.218585 6243 ts_tablet_manager.cc:1428] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:14:40.218600 6233 heartbeater.cc:503] Master 127.21.204.126:43839 was elected leader, sending a full tablet report...
I20260923 15:14:40.218681 6101 heartbeater.cc:503] Master 127.21.204.126:43839 was elected leader, sending a full tablet report...
I20260923 15:14:40.218585 6242 ts_tablet_manager.cc:1428] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260923 15:14:40.280192 6102 tablet.cc:2369] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:14:40.322896 6247 raft_consensus.cc:482] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:14:40.323064 6247 raft_consensus.cc:504] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.323472 6247 leader_election.cc:288] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957), 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
I20260923 15:14:40.326357 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.323724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:53237 (local address 127.21.204.67:41957)
0923 15:14:40.323775 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.323778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.323788 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:14:40.323816 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.323817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.323828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.323889 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:40.324085 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.324599 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.325541 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.325986 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.326077 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.326087 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.326255 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.326293 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:14:40.326357 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.323613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:41957 (local address 127.21.204.66:53237)
0923 15:14:40.323673 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.323684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:40.323727 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:40.323947 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:40.323948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:40.323950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:40.324007 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.324009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.324699 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.324700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.325441 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.325443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.326077 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.326078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.326170 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.326173 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:14:40.326236 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 15:14:40.326278 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 15:14:40.326632 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.323660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36035 (local address 127.21.204.66:48171)
0923 15:14:40.323923 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.323933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:14:40.324000 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:14:40.324157 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:14:40.324158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:14:40.324160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:14:40.324259 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.324261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.324992 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.324993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.325718 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:14:40.325720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.326401 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.326402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:14:40.326468 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.326471 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:14:40.326515 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 15:14:40.326554 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":162,"threads_started":1}
I20260923 15:14:40.326637 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.323716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:48171 (local address 127.21.204.65:36035)
0923 15:14:40.323762 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.323765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.323999 (+ 234us) server_negotiation.cc:407] Connection header received
0923 15:14:40.324062 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.324063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.324091 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.324155 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:14:40.324329 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.324900 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.325834 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.326304 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.326400 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.326479 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.326530 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.326565 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:14:40.326901 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacadb262746463aabc79dbd17b10607" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" is_pre_election: true
I20260923 15:14:40.327004 5915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacadb262746463aabc79dbd17b10607" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278b140ae0364a2784e69a176dfaca7a" is_pre_election: true
I20260923 15:14:40.327080 6187 raft_consensus.cc:2454] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 0.
I20260923 15:14:40.327185 5915 raft_consensus.cc:2454] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 0.
I20260923 15:14:40.327335 5988 leader_election.cc:302] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a39f8c0ebe7744cea8b85e9585cb8630, c5d76a86be1a4265a759d84faa293651; no voters:
I20260923 15:14:40.327445 6247 raft_consensus.cc:2781] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:14:40.327497 6247 raft_consensus.cc:482] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:14:40.327523 6247 raft_consensus.cc:3037] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:40.328229 6247 raft_consensus.cc:504] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.328425 6247 leader_election.cc:288] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 election: Requested vote from peers a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957), 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
I20260923 15:14:40.328521 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacadb262746463aabc79dbd17b10607" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39f8c0ebe7744cea8b85e9585cb8630"
I20260923 15:14:40.328508 5915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacadb262746463aabc79dbd17b10607" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278b140ae0364a2784e69a176dfaca7a"
I20260923 15:14:40.328591 6187 raft_consensus.cc:3037] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:40.328598 5915 raft_consensus.cc:3037] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:14:40.329210 6187 raft_consensus.cc:2454] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 1.
I20260923 15:14:40.329276 5915 raft_consensus.cc:2454] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 1.
I20260923 15:14:40.329396 5988 leader_election.cc:302] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a39f8c0ebe7744cea8b85e9585cb8630, c5d76a86be1a4265a759d84faa293651; no voters:
I20260923 15:14:40.329515 6247 raft_consensus.cc:2781] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:14:40.329682 6247 raft_consensus.cc:686] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 LEADER]: Becoming Leader. State: Replica: c5d76a86be1a4265a759d84faa293651, State: Running, Role: LEADER
I20260923 15:14:40.329788 6247 consensus_queue.cc:237] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:14:40.330709 5771 catalog_manager.cc:5495] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5d76a86be1a4265a759d84faa293651 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "c5d76a86be1a4265a759d84faa293651" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } health_report { overall_health: UNKNOWN } } }
I20260923 15:14:40.354898 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.352648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.21.204.66:39273)
0923 15:14:40.352699 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:14:40.352701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:14:40.352708 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:14:40.352743 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:14:40.352744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:14:40.352762 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:14:40.352828 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:14:40.352928 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.353374 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.354161 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:14:40.354362 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:14:40.354435 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:14:40.354597 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:14:40.354696 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:14:40.354761 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 15:14:40.354797 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 15:14:40.354816 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:14:40.356840 6187 raft_consensus.cc:1261] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Refusing update from remote peer c5d76a86be1a4265a759d84faa293651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:40.356845 5915 raft_consensus.cc:1261] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Refusing update from remote peer c5d76a86be1a4265a759d84faa293651: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:14:40.357296 6249 consensus_queue.cc:1035] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:40.357475 6247 consensus_queue.cc:1035] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:14:40.359308 6251 mvcc.cc:205] Tried to move back new op lower bound from 7332562863538774016 to 7332562863431643136. Current Snapshot: MvccSnapshot[applied={T|T < 7332562863538774016}]
I20260923 15:14:40.360044 6253 mvcc.cc:205] Tried to move back new op lower bound from 7332562863538774016 to 7332562863431643136. Current Snapshot: MvccSnapshot[applied={T|T < 7332562863538774016}]
I20260923 15:14:40.361824 5772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:40.361955 5772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:14:40.362692 5772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Sending DeleteTablet for 3 replicas of tablet cacadb262746463aabc79dbd17b10607
I20260923 15:14:40.363104 6034 tablet_service.cc:1514] Processing DeleteTablet for tablet cacadb262746463aabc79dbd17b10607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:40 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:47340
I20260923 15:14:40.363147 5894 tablet_service.cc:1514] Processing DeleteTablet for tablet cacadb262746463aabc79dbd17b10607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:40 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:32768
I20260923 15:14:40.363121 6167 tablet_service.cc:1514] Processing DeleteTablet for tablet cacadb262746463aabc79dbd17b10607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:14:40 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:42996
I20260923 15:14:40.363242 5970 tablet_replica.cc:330] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: stopping tablet replica
I20260923 15:14:40.363300 5970 raft_consensus.cc:2229] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:40.363340 5970 raft_consensus.cc:2258] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:40.363405 6261 tablet_replica.cc:330] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: stopping tablet replica
I20260923 15:14:40.363463 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5810
I20260923 15:14:40.363504 6261 raft_consensus.cc:2229] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:14:40.363750 5970 ts_tablet_manager.cc:1905] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:40.364040 6261 raft_consensus.cc:2258] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:40.364435 6261 ts_tablet_manager.cc:1905] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:14:40.366184 6261 ts_tablet_manager.cc:1918] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:14:40.366317 6261 log.cc:1196] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/cacadb262746463aabc79dbd17b10607
I20260923 15:14:40.366671 6261 ts_tablet_manager.cc:1939] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Deleting consensus metadata
I20260923 15:14:40.367151 5758 catalog_manager.cc:4841] TS a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957): tablet cacadb262746463aabc79dbd17b10607 (table test-table [id=53e3c71e8f44440eb5e5369a2af3dc24]) successfully deleted
I20260923 15:14:40.370044 6260 tablet_replica.cc:330] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: stopping tablet replica
I20260923 15:14:40.370158 6260 raft_consensus.cc:2229] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 LEADER]: Raft consensus shutting down.
W20260923 15:14:40.370152 5759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:36035 (error 108)
I20260923 15:14:40.370258 6260 raft_consensus.cc:2258] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:14:40.370491 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5966
I20260923 15:14:40.372167 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.371980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36035 (local address 127.0.0.1:32772)
0923 15:14:40.372087 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.372139 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:36035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260923 15:14:40.372295 5759 catalog_manager.cc:4584] TS 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cacadb262746463aabc79dbd17b10607: Network error: Client connection negotiation failed: client connection to 127.21.204.65:36035: connect: Connection refused (error 111)
I20260923 15:14:40.375514 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 6104
I20260923 15:14:40.375599 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:14:40.375462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:39273 (local address 127.0.0.1:47356)
0923 15:14:40.375520 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:14:40.375547 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.66:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 15:14:40.380470 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 5739
I20260923 15:15:00.386102 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43839
--webserver_interface=127.21.204.126
--webserver_port=37953
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:43839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:00.468852 6263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:00.469061 6263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:00.469085 6263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:00.470256 6263 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 15:15:00.470295 6263 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 15:15:00.470309 6263 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 15:15:00.470320 6263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:00.470333 6263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:00.470382 6263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:00.470400 6263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:00.470423 6263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:00.471922 6263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:43839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=37953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:00.472260 6263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:00.472530 6263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:00.477600 6268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:00.477613 6263 server_base.cc:1034] running on GCE node
W20260923 15:15:00.477813 6271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:00.477826 6269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:00.478118 6263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:00.478418 6263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:00.479580 6263 hybrid_clock.cc:648] HybridClock initialized: now 1790176500479574 us; error 34 us; skew 500 ppm
Sep 23 15:15:00 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:00.482085 6263 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:00.482566 6263 webserver.cc:460] Webserver started at http://127.21.204.126:37953/ using document root <none> and password file <none>
I20260923 15:15:00.482877 6263 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:00.482947 6263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:00.484328 6263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 15:15:00.484942 6278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:00.485133 6263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:00.485221 6263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "8db893b01b4641eebfd92236e08e1a7c"
format_stamp: "Formatted at 2026-09-23 15:14:18 on dist-test-slave-t5lt"
server_key: "45068a22a6eb207dc76e8a4209ba4c5b"
server_key_iv: "77c37771d546e214d167a86988b6767d"
server_key_version: "encryptionkey@0"
I20260923 15:15:00.485548 6263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:00.499588 6263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:00.501572 6263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:00.501716 6263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:00.505546 6263 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:43839
I20260923 15:15:00.505623 6330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:43839 every 8 connection(s)
I20260923 15:15:00.506029 6263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:00.507464 6331 sys_catalog.cc:263] Verifying existing consensus state
I20260923 15:15:00.508057 6331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Bootstrap starting.
I20260923 15:15:00.511204 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 6263
I20260923 15:15:00.511564 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:36035
--local_ip_for_outbound_sockets=127.21.204.65
--tserver_master_addrs=127.21.204.126:43839
--webserver_port=43939
--webserver_interface=127.21.204.65
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:15:00.516978 6331 log.cc:824] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:00.519209 6331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:15:00.519516 6331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Bootstrap complete.
I20260923 15:15:00.521714 6331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:15:00.521983 6331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db893b01b4641eebfd92236e08e1a7c, State: Initialized, Role: FOLLOWER
I20260923 15:15:00.522158 6331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:15:00.522241 6331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:00.522306 6331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:00.522354 6331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 2 FOLLOWER]: Advancing to term 3
I20260923 15:15:00.523308 6331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:15:00.523415 6331 leader_election.cc:302] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8db893b01b4641eebfd92236e08e1a7c; no voters:
I20260923 15:15:00.523588 6331 leader_election.cc:288] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 15:15:00.523651 6335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 3 FOLLOWER]: Leader election won for term 3
I20260923 15:15:00.523805 6335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [term 3 LEADER]: Becoming Leader. State: Replica: 8db893b01b4641eebfd92236e08e1a7c, State: Running, Role: LEADER
I20260923 15:15:00.523852 6331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:00.523908 6335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } }
I20260923 15:15:00.524147 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8db893b01b4641eebfd92236e08e1a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } } }
I20260923 15:15:00.524232 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:00.524143 6335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db893b01b4641eebfd92236e08e1a7c. Latest consensus state: current_term: 3 leader_uuid: "8db893b01b4641eebfd92236e08e1a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db893b01b4641eebfd92236e08e1a7c" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43839 } } }
I20260923 15:15:00.524389 6335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:00.524492 6340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:00.525461 6340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:00.525560 6340 catalog_manager.cc:1259] Loaded cluster ID: 918431f34c9f492a8ef90b9cdf27e344
I20260923 15:15:00.525593 6340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:00.526997 6340 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:00.527179 6340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Loaded TSK: 0
I20260923 15:15:00.527230 6340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8db893b01b4641eebfd92236e08e1a7c: Loaded TSK: 1
F20260923 15:15:00.527470 6340 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faab954b9ad google::LogMessage::Fail() at ??:0
@ 0x7faab954e88a google::LogMessage::SendToLog() at ??:0
@ 0x7faab954b4de google::LogMessage::Flush() at ??:0
@ 0x7faab954ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faabbcb8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faabbcb5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faac14a51cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faac14a548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faac14a5594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7faac146bc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7faac1475ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faabb4190d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faabb40e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faabc2726db start_thread at ??:0
@ 0x7faab804961f clone at ??:0
W20260923 15:15:00.611881 6333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:00.612077 6333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:00.612100 6333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:00.613260 6333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:00.613302 6333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:00.613329 6333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:15:00.614837 6333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:36035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=43939
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:00.615077 6333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:00.615309 6333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:00.620292 6372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:00.620373 6333 server_base.cc:1034] running on GCE node
W20260923 15:15:00.620437 6375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:00.620275 6373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:00.620711 6333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:00.620967 6333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:00.622140 6333 hybrid_clock.cc:648] HybridClock initialized: now 1790176500622124 us; error 33 us; skew 500 ppm
Sep 23 15:15:00 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:00.624818 6333 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:15:00.625339 6333 webserver.cc:460] Webserver started at http://127.21.204.65:43939/ using document root <none> and password file <none>
I20260923 15:15:00.625579 6333 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:00.625644 6333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:00.627031 6333 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:00.627805 6382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:00.627988 6333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:00.628090 6333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "278b140ae0364a2784e69a176dfaca7a"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "6667a4b96d814e34225c0b9426a017e6"
server_key_iv: "d4cd52c9be44f402aeb600dbb913a0bc"
server_key_version: "encryptionkey@0"
I20260923 15:15:00.628433 6333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:00.660714 6333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:00.663156 6333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:00.663343 6333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:00.663628 6333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:00.664175 6389 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 15:15:00.666204 6333 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 15:15:00.666280 6333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 15:15:00.666335 6333 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 15:15:00.667145 6333 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 15:15:00.667212 6333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 15:15:00.667268 6389 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap starting.
I20260923 15:15:00.669171 6389 log.cc:824] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:00.672096 6389 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:15:00.672406 6389 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Bootstrap complete.
I20260923 15:15:00.672551 6389 ts_tablet_manager.cc:1397] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 15:15:00.674489 6389 raft_consensus.cc:348] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:00.674880 6389 raft_consensus.cc:729] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278b140ae0364a2784e69a176dfaca7a, State: Initialized, Role: FOLLOWER
I20260923 15:15:00.675088 6389 consensus_queue.cc:260] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:00.675446 6389 ts_tablet_manager.cc:1428] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 15:15:00.676028 6333 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:36035
I20260923 15:15:00.676070 6498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:36035 every 8 connection(s)
I20260923 15:15:00.676407 6333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:15:00.677358 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 6333
I20260923 15:15:00.677738 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:39273
--local_ip_for_outbound_sockets=127.21.204.66
--tserver_master_addrs=127.21.204.126:43839
--webserver_port=41339
--webserver_interface=127.21.204.66
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:00.717455 6501 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:00.677094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:49623)
0923 15:15:00.677313 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.677368 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:00.677560 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:00.717194 (+ 39634us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
W20260923 15:15:00.717820 6499 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:43839 (0 consecutive failures): Network error: Failed to ping master at 127.21.204.126:43839: Client connection negotiation failed: client connection to 127.21.204.126:43839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 15:15:00.718254 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.718099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:47825)
0923 15:15:00.718166 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.718222 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 15:15:00.718737 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.718623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:58559)
0923 15:15:00.718673 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.718710 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 15:15:00.718891 6499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 15:15:00.763546 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:00.763738 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:00.763764 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:00.764912 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:00.764953 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:00.764981 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:15:00.766525 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:39273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=41339
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:00.766764 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:00.766993 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:00.772104 6510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:00.772102 6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:00.772097 6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:00.772691 6502 server_base.cc:1034] running on GCE node
I20260923 15:15:00.772910 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:00.773152 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:00.774295 6502 hybrid_clock.cc:648] HybridClock initialized: now 1790176500774283 us; error 28 us; skew 500 ppm
Sep 23 15:15:00 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:00.776780 6502 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:15:00.777186 6502 webserver.cc:460] Webserver started at http://127.21.204.66:41339/ using document root <none> and password file <none>
I20260923 15:15:00.777415 6502 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:00.777479 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:00.778865 6502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:00.779554 6517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:00.779743 6502 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:15:00.779841 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "c5d76a86be1a4265a759d84faa293651"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "dd95f8ff529e3c8ea662ddd2dce1b204"
server_key_iv: "054bb47182645de05c68841495615110"
server_key_version: "encryptionkey@0"
I20260923 15:15:00.780172 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:00.796939 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:00.798828 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:00.798983 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:00.799207 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:00.799612 6524 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 15:15:00.801016 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 15:15:00.801097 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 15:15:00.801180 6502 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 15:15:00.801848 6502 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 15:15:00.801927 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:00.801997 6524 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Bootstrap starting.
I20260923 15:15:00.803735 6524 log.cc:824] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:00.807192 6524 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:15:00.807562 6524 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Bootstrap complete.
I20260923 15:15:00.807749 6524 ts_tablet_manager.cc:1397] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 15:15:00.809515 6502 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:39273
I20260923 15:15:00.809544 6632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:39273 every 8 connection(s)
I20260923 15:15:00.809971 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:15:00.810055 6524 raft_consensus.cc:348] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:00.810515 6524 raft_consensus.cc:729] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d76a86be1a4265a759d84faa293651, State: Initialized, Role: FOLLOWER
I20260923 15:15:00.810688 6524 consensus_queue.cc:260] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:00.810963 6524 ts_tablet_manager.cc:1428] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260923 15:15:00.811159 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.810763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:38541)
0923 15:15:00.810924 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.811008 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260923 15:15:00.811370 6633 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:43839 (0 consecutive failures): Network error: Failed to ping master at 127.21.204.126:43839: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
I20260923 15:15:00.811640 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.811542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:49467)
0923 15:15:00.811588 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.811619 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 15:15:00.811906 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.811828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:38805)
0923 15:15:00.811867 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.811891 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 15:15:00.812009 6633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 15:15:00.813071 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 6502
I20260923 15:15:00.813337 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:41957
--local_ip_for_outbound_sockets=127.21.204.67
--tserver_master_addrs=127.21.204.126:43839
--webserver_port=34453
--webserver_interface=127.21.204.67
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:00.899441 6637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:00.899616 6637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:00.899639 6637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:00.901134 6637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:00.901185 6637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:00.901211 6637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:15:00.902721 6637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:41957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=34453
--tserver_master_addrs=127.21.204.126:43839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:00.902967 6637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:00.903179 6637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:00.908051 6643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:00.908039 6642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:00.908030 6645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:00.908056 6637 server_base.cc:1034] running on GCE node
I20260923 15:15:00.908459 6637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:00.908689 6637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:00.909860 6637 hybrid_clock.cc:648] HybridClock initialized: now 1790176500909839 us; error 27 us; skew 500 ppm
Sep 23 15:15:00 dist-test-slave-t5lt krb5kdc[5191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:00.918138 6637 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:15:00.918619 6637 webserver.cc:460] Webserver started at http://127.21.204.67:34453/ using document root <none> and password file <none>
I20260923 15:15:00.918865 6637 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:00.918912 6637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:00.920265 6637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:00.920869 6652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:00.921156 6637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:15:00.921253 6637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "a39f8c0ebe7744cea8b85e9585cb8630"
format_stamp: "Formatted at 2026-09-23 15:14:19 on dist-test-slave-t5lt"
server_key: "150b0e20ed8afe74f73d5df359f62c49"
server_key_iv: "bfe0f4872afd90511b79eea024e9d281"
server_key_version: "encryptionkey@0"
I20260923 15:15:00.921593 6637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:00.931869 6637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:00.933697 6637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:00.933844 6637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:00.934124 6637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:00.934422 6637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:00.934471 6637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:00.934512 6637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:00.934540 6637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:00.940502 6637 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:41957
I20260923 15:15:00.940549 6765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:41957 every 8 connection(s)
I20260923 15:15:00.940878 6637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:15:00.941946 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.941538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:55623)
0923 15:15:00.941681 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.941746 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260923 15:15:00.942456 6766 heartbeater.cc:644] Failed to heartbeat to 127.21.204.126:43839 (0 consecutive failures): Network error: Failed to ping master at 127.21.204.126:43839: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
I20260923 15:15:00.942744 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.942659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:49517)
0923 15:15:00.942701 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.942727 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 15:15:00.942986 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:00.942931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:37131)
0923 15:15:00.942955 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:00.942973 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260923 15:15:00.943094 6766 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 15:15:00.948275 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 6637
I20260923 15:15:01.719553 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.719312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:48795)
0923 15:15:01.719441 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:01.719487 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260923 15:15:01.812621 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.812397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:36249)
0923 15:15:01.812535 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:01.812578 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260923 15:15:01.943638 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.943413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:55779)
0923 15:15:01.943546 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:01.943587 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260923 15:15:01.962690 6773 raft_consensus.cc:482] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:15:01.962815 6773 raft_consensus.cc:504] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:01.963105 6773 leader_election.cc:288] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957), 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
Sep 23 15:15:01 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.67@KRBTEST.COM
Sep 23 15:15:01 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.65@KRBTEST.COM
I20260923 15:15:01.972957 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.963296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:41957 (local address 127.21.204.66:42081)
0923 15:15:01.963488 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:01.963498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:01.963575 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:01.964246 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:01.964247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:01.964442 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:01.964569 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:01.964575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.965648 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.965649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:01.966504 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:01.966507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.967086 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.967087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:01.967211 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:01.967791 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:01.967800 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:01.969274 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:01.971493 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:01.971501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:01.971503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:01.971719 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:01.971971 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:01.971971 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:01.971972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:01.972077 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:01.972450 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:01.972457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:01.972584 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 15:15:01.972685 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 15:15:01.972851 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":79,"thread_start_us":75,"threads_started":1}
I20260923 15:15:01.972970 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.963443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:42081 (local address 127.21.204.67:41957)
0923 15:15:01.963580 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:01.963590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:01.963620 (+ 30us) server_negotiation.cc:407] Connection header received
0923 15:15:01.964094 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:01.964097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:01.964130 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:01.964260 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:01.964692 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.965545 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.966595 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.966996 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.967122 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:01.969381 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:01.969385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:01.969386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:01.969400 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:01.971397 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:01.971807 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:01.971810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:01.971810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:01.971898 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:01.972141 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:01.972142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:01.972143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:01.972372 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:01.972458 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:01.972740 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 15:15:01.972865 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":41,"threads_started":1}
I20260923 15:15:01.973230 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.963295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:36035 (local address 127.21.204.66:54789)
0923 15:15:01.963370 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:01.963401 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:01.963575 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:01.963900 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:01.963905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:01.964218 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:01.964597 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:01.964600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.965675 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.965682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:01.966504 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:01.966507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.967095 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.967096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:01.967216 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:01.967847 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:01.967852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:01.969708 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:01.972021 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:01.972022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:01.972023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:01.972194 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:01.972457 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:01.972458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:01.972458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:01.972518 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:01.972888 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:01.972889 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:01.972934 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 15:15:01.973015 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 15:15:01.973120 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":37}
I20260923 15:15:01.973932 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.963473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:54789 (local address 127.21.204.65:36035)
0923 15:15:01.963650 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:01.963654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:01.963665 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:15:01.963744 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:01.963746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:01.963765 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:01.963865 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:01.964725 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.965558 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.966595 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:01.966996 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:01.967118 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:01.969838 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:01.969842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:01.969842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:01.969851 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:01.971934 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:01.972287 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:01.972290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:01.972290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:01.972388 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:01.972584 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:01.972584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:01.972585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:01.972810 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:01.972900 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:01.973646 (+ 746us) server_negotiation.cc:299] Negotiation successful
0923 15:15:01.973784 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
W20260923 15:15:01.974014 6520 leader_election.cc:341] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957): Not found: Tablet not found: cacadb262746463aabc79dbd17b10607
I20260923 15:15:01.974390 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacadb262746463aabc79dbd17b10607" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "278b140ae0364a2784e69a176dfaca7a" is_pre_election: true
I20260923 15:15:01.974514 6447 raft_consensus.cc:2454] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 1.
I20260923 15:15:01.974736 6521 leader_election.cc:302] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 278b140ae0364a2784e69a176dfaca7a, c5d76a86be1a4265a759d84faa293651; no voters: a39f8c0ebe7744cea8b85e9585cb8630
I20260923 15:15:01.974879 6773 raft_consensus.cc:2781] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 15:15:01.974964 6773 raft_consensus.cc:482] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:15:01.975001 6773 raft_consensus.cc:3037] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 1 FOLLOWER]: Advancing to term 2
I20260923 15:15:01.976202 6773 raft_consensus.cc:504] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:01.976356 6773 leader_election.cc:288] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 2 election: Requested vote from peers a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957), 278b140ae0364a2784e69a176dfaca7a (127.21.204.65:36035)
I20260923 15:15:01.976512 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cacadb262746463aabc79dbd17b10607" candidate_uuid: "c5d76a86be1a4265a759d84faa293651" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "278b140ae0364a2784e69a176dfaca7a"
I20260923 15:15:01.976600 6447 raft_consensus.cc:3037] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 1 FOLLOWER]: Advancing to term 2
W20260923 15:15:01.976625 6520 leader_election.cc:341] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957): Not found: Tablet not found: cacadb262746463aabc79dbd17b10607
I20260923 15:15:01.977707 6447 raft_consensus.cc:2454] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d76a86be1a4265a759d84faa293651 in term 2.
I20260923 15:15:01.977907 6521 leader_election.cc:302] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 278b140ae0364a2784e69a176dfaca7a, c5d76a86be1a4265a759d84faa293651; no voters: a39f8c0ebe7744cea8b85e9585cb8630
I20260923 15:15:01.978008 6773 raft_consensus.cc:2781] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 15:15:01.978132 6773 raft_consensus.cc:686] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [term 2 LEADER]: Becoming Leader. State: Replica: c5d76a86be1a4265a759d84faa293651, State: Running, Role: LEADER
I20260923 15:15:01.978251 6773 consensus_queue.cc:237] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:01.978503 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:01.978419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:39225)
0923 15:15:01.978450 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:01.978476 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260923 15:15:02.370159 6520 consensus_peers.cc:487] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 -> Peer a39f8c0ebe7744cea8b85e9585cb8630 (127.21.204.67:41957): Couldn't send request to peer a39f8c0ebe7744cea8b85e9585cb8630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cacadb262746463aabc79dbd17b10607. This is attempt 1: this message will repeat every 5th retry.
I20260923 15:15:02.591847 6447 raft_consensus.cc:1261] T cacadb262746463aabc79dbd17b10607 P 278b140ae0364a2784e69a176dfaca7a [term 2 FOLLOWER]: Refusing update from remote peer c5d76a86be1a4265a759d84faa293651: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260923 15:15:02.592296 6773 consensus_queue.cc:1035] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651 [LEADER]: Connected to new peer: Peer: permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260923 15:15:02.593279 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.593099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:45621)
0923 15:15:02.593181 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:02.593238 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":42,"threads_started":1}
I20260923 15:15:02.594255 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.593985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:42063)
0923 15:15:02.594194 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:02.594223 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":55,"threads_started":1}
I20260923 15:15:02.594245 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.594135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:37863)
0923 15:15:02.594186 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:02.594221 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 15:15:02.944356 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.944111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:57577)
0923 15:15:02.944274 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:02.944310 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260923 15:15:02.971691 6787 ts_tablet_manager.cc:927] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Initiating tablet copy from peer c5d76a86be1a4265a759d84faa293651 (127.21.204.66:39273)
I20260923 15:15:02.971982 6787 tablet_copy_client.cc:316] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: tablet copy: Beginning tablet copy session from remote peer at address 127.21.204.66:39273
Sep 23 15:15:02 dist-test-slave-t5lt krb5kdc[5191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176500, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.66@KRBTEST.COM
I20260923 15:15:02.980139 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.972303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:39273 (local address 127.21.204.67:57593)
0923 15:15:02.972364 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:02.972401 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:02.972473 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:02.972681 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:02.972685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:02.972963 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:02.973024 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:02.973025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:02.974098 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:02.974099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:02.974687 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:02.974688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:02.975102 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:02.975103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:02.975162 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:02.975622 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:02.975627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:02.977052 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:02.978970 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:02.978978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:02.978980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:02.979184 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:02.979389 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:02.979389 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:02.979390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:02.979446 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:02.979788 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:02.979797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:02.979852 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 15:15:02.979937 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:15:02.980057 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 15:15:02.980151 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.972402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:57593 (local address 127.21.204.66:39273)
0923 15:15:02.972552 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:02.972556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:02.972566 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:15:02.972590 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:02.972592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:02.972604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:02.972669 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:02.973094 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:02.974011 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:02.974755 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:02.975038 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:02.975092 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:02.977135 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:02.977144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:02.977145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:02.977151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:02.978878 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:02.979267 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:02.979270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:02.979270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:02.979328 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:02.979524 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:02.979525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:02.979525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:02.979685 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:02.979767 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:02.980015 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 15:15:02.980084 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260923 15:15:02.980666 6607 tablet_copy_service.cc:140] P c5d76a86be1a4265a759d84faa293651: Received BeginTabletCopySession request for tablet cacadb262746463aabc79dbd17b10607 from peer a39f8c0ebe7744cea8b85e9585cb8630 ({username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:57593)
I20260923 15:15:02.980742 6607 tablet_copy_service.cc:161] P c5d76a86be1a4265a759d84faa293651: Beginning new tablet copy session on tablet cacadb262746463aabc79dbd17b10607 from peer a39f8c0ebe7744cea8b85e9585cb8630 at {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:57593: session id = a39f8c0ebe7744cea8b85e9585cb8630-cacadb262746463aabc79dbd17b10607
I20260923 15:15:02.981320 6607 tablet_copy_source_session.cc:205] T cacadb262746463aabc79dbd17b10607 P c5d76a86be1a4265a759d84faa293651: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 15:15:02.982447 6787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacadb262746463aabc79dbd17b10607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:02.985180 6787 tablet_copy_client.cc:791] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: tablet copy: Starting download of 0 data blocks...
I20260923 15:15:02.985347 6787 tablet_copy_client.cc:655] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: tablet copy: Starting download of 1 WAL segments...
I20260923 15:15:02.986243 6787 tablet_copy_client.cc:531] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 15:15:02.986907 6787 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Bootstrap starting.
I20260923 15:15:02.988234 6787 log.cc:824] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:02.990618 6787 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 15:15:02.990841 6787 tablet_bootstrap.cc:492] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Bootstrap complete.
I20260923 15:15:02.990975 6787 ts_tablet_manager.cc:1397] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260923 15:15:02.991876 6787 raft_consensus.cc:348] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:02.992043 6787 raft_consensus.cc:729] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39f8c0ebe7744cea8b85e9585cb8630, State: Initialized, Role: FOLLOWER
I20260923 15:15:02.992233 6787 consensus_queue.cc:260] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d76a86be1a4265a759d84faa293651" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39273 } } peers { permanent_uuid: "a39f8c0ebe7744cea8b85e9585cb8630" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 41957 } } peers { permanent_uuid: "278b140ae0364a2784e69a176dfaca7a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 36035 } }
I20260923 15:15:02.992537 6787 ts_tablet_manager.cc:1428] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 15:15:02.992822 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.992725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:57029)
0923 15:15:02.992765 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:02.992795 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 15:15:02.992835 6607 tablet_copy_service.cc:342] P c5d76a86be1a4265a759d84faa293651: Request end of tablet copy session a39f8c0ebe7744cea8b85e9585cb8630-cacadb262746463aabc79dbd17b10607 received from {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:57593
I20260923 15:15:02.992888 6607 tablet_copy_service.cc:433] P c5d76a86be1a4265a759d84faa293651: ending tablet copy session a39f8c0ebe7744cea8b85e9585cb8630-cacadb262746463aabc79dbd17b10607 on tablet cacadb262746463aabc79dbd17b10607 with peer a39f8c0ebe7744cea8b85e9585cb8630
I20260923 15:15:02.993202 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:02.993042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:35955)
0923 15:15:02.993129 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:02.993153 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 15:15:03.488188 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:03.488001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:57201)
0923 15:15:03.488095 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:03.488129 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
I20260923 15:15:03.488191 6720 raft_consensus.cc:1203] T cacadb262746463aabc79dbd17b10607 P a39f8c0ebe7744cea8b85e9585cb8630 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260923 15:15:03.488649 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:03.488550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:37399)
0923 15:15:03.488594 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:03.488625 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 15:15:03.594830 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:03.594625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:42513)
0923 15:15:03.594727 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:03.594781 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260923 15:15:04.488871 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:04.488585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:56623)
0923 15:15:04.488784 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:04.488822 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":57,"threads_started":1}
I20260923 15:15:04.489241 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:04.489057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:46741)
0923 15:15:04.489170 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:04.489203 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260923 15:15:04.595472 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:04.595253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:51617)
0923 15:15:04.595396 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:04.595424 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260923 15:15:05.489588 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:05.489342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:41745)
0923 15:15:05.489494 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:05.489537 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260923 15:15:05.489959 6800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:05.489737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:41009)
0923 15:15:05.489831 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:05.489866 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":51,"threads_started":1}
I20260923 15:15:05.596172 6801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:05.595945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:48679)
0923 15:15:05.596076 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:05.596122 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260923 15:15:06.490386 6802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:06.490164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:50845)
0923 15:15:06.490300 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:06.490338 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260923 15:15:06.490603 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:06.490414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:53249)
0923 15:15:06.490519 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:06.490552 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
I20260923 15:15:06.491001 6802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:06.490883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:45483)
0923 15:15:06.490939 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:06.490973 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 15:15:06.596861 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:06.596641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:51553)
0923 15:15:06.596780 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:06.596816 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260923 15:15:07.491432 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:07.491170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:50097)
0923 15:15:07.491334 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:07.491382 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260923 15:15:07.491628 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:07.491411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:52671)
0923 15:15:07.491541 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:07.491579 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260923 15:15:07.597697 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:07.597383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:44557)
0923 15:15:07.597582 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:07.597626 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260923 15:15:08.492074 6809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:08.491857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:43289)
0923 15:15:08.491987 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:08.492021 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260923 15:15:08.492297 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:08.492031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:34889)
0923 15:15:08.492184 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:08.492253 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260923 15:15:08.598518 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:08.598283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:38713)
0923 15:15:08.598433 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:08.598470 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260923 15:15:09.492956 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:09.492695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:41387)
0923 15:15:09.492859 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:09.492907 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":60,"threads_started":1}
I20260923 15:15:09.492985 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:09.492750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:48407)
0923 15:15:09.492910 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:09.492944 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260923 15:15:09.599236 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:09.599016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:45239)
0923 15:15:09.599147 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:09.599183 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260923 15:15:10.493634 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:10.493413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:51219)
0923 15:15:10.493551 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:10.493584 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260923 15:15:10.493670 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:10.493455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:34965)
0923 15:15:10.493595 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:10.493627 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260923 15:15:10.599989 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:10.599755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:47371)
0923 15:15:10.599899 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:10.599939 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260923 15:15:11.494474 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:11.494169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:36185)
0923 15:15:11.494316 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:11.494344 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260923 15:15:11.494474 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:11.494165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:37317)
0923 15:15:11.494348 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:11.494381 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260923 15:15:11.600703 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:11.600476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:33697)
0923 15:15:11.600620 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:11.600655 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260923 15:15:12.495262 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:12.495020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:56185)
0923 15:15:12.495178 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:12.495211 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260923 15:15:12.495263 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:12.495020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:37147)
0923 15:15:12.495180 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:12.495218 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260923 15:15:12.601495 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:12.601247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:44455)
0923 15:15:12.601406 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:12.601445 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260923 15:15:13.495980 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:13.495749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:39453)
0923 15:15:13.495907 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:13.495935 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260923 15:15:13.495980 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:13.495748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:48569)
0923 15:15:13.495895 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:13.495933 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260923 15:15:13.602279 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:13.602049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:50267)
0923 15:15:13.602194 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:13.602231 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260923 15:15:14.496712 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:14.496517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:34309)
0923 15:15:14.496626 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:14.496664 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260923 15:15:14.496797 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:14.496575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:57409)
0923 15:15:14.496709 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:14.496744 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260923 15:15:14.602962 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:14.602765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:38171)
0923 15:15:14.602881 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:14.602914 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260923 15:15:15.497452 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:15.497191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.67:38977)
0923 15:15:15.497371 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:15.497407 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260923 15:15:15.497452 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:15.497190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.66:58113)
0923 15:15:15.497365 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:15.497403 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260923 15:15:15.603744 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:15.603484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43839 (local address 127.21.204.65:58171)
0923 15:15:15.603637 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:15.603679 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.126:43839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260923 15:15:15.952785 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 6333
I20260923 15:15:15.958092 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 6502
I20260923 15:15:15.962774 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 6637
2026-09-23T15:15:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58225 ms)
[----------] 30 tests from SecurityITest (237561 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:16 dist-test-slave-t5lt krb5kdc[6836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:16 dist-test-slave-t5lt krb5kdc[6836](info): set up 2 sockets
Sep 23 15:15:16 dist-test-slave-t5lt krb5kdc[6836](info): commencing operation
krb5kdc: starting...
W20260923 15:15:17.003479 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:17.124320 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:46847
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:46847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:17.205977 6850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:17.206183 6850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:17.206205 6850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:17.207355 6850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:17.207396 6850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:17.207412 6850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:17.207427 6850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:17.207438 6850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:17.208917 6850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:46847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:46847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:17.209167 6850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:17.209401 6850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:17.214644 6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:17.214685 6856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:17.214774 6850 server_base.cc:1034] running on GCE node
W20260923 15:15:17.214684 6855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:17.215102 6850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:17.215463 6850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:17.216665 6850 hybrid_clock.cc:648] HybridClock initialized: now 1790176517216638 us; error 37 us; skew 500 ppm
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:17.219571 6850 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:17.220026 6850 webserver.cc:460] Webserver started at http://127.21.204.126:45543/ using document root <none> and password file <none>
I20260923 15:15:17.220227 6850 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:17.220275 6850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:17.220374 6850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:17.221459 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "0fabd4fbd7384405b4657f3eb4b066ba"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "28b66d589fac92a0c70f4295a3ad8ab6"
server_key_iv: "9b6d1c6d04ad756c4b3ea6c0cae79622"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.221812 6850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "0fabd4fbd7384405b4657f3eb4b066ba"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "28b66d589fac92a0c70f4295a3ad8ab6"
server_key_iv: "9b6d1c6d04ad756c4b3ea6c0cae79622"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.223316 6850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:17.224088 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.224265 6850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:15:17.224347 6850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "0fabd4fbd7384405b4657f3eb4b066ba"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "28b66d589fac92a0c70f4295a3ad8ab6"
server_key_iv: "9b6d1c6d04ad756c4b3ea6c0cae79622"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.224398 6850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:17.236553 6850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:17.238332 6850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:17.238485 6850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:17.242339 6850 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:46847
I20260923 15:15:17.242391 6917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:46847 every 8 connection(s)
I20260923 15:15:17.242719 6850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:17.243606 6918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:17.245818 6918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba: Bootstrap starting.
I20260923 15:15:17.246413 6918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:17.246711 6918 log.cc:824] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:17.247442 6918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba: No bootstrap required, opened a new log
I20260923 15:15:17.248708 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 6850
I20260923 15:15:17.248831 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:17.249032 22321 external_mini_cluster.cc:1428] Setting key 029c4772b586b88aed2568bf8987a09c
I20260923 15:15:17.249581 6918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46847 } }
I20260923 15:15:17.249771 6918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:17.249800 6918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fabd4fbd7384405b4657f3eb4b066ba, State: Initialized, Role: FOLLOWER
I20260923 15:15:17.249994 6918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46847 } }
I20260923 15:15:17.250094 6918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:17.250130 6918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:17.250216 6918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:17.251284 6918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46847 } }
I20260923 15:15:17.251377 6918 leader_election.cc:302] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fabd4fbd7384405b4657f3eb4b066ba; no voters:
I20260923 15:15:17.251579 6918 leader_election.cc:288] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:17.251601 6923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:17.251709 6923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [term 1 LEADER]: Becoming Leader. State: Replica: 0fabd4fbd7384405b4657f3eb4b066ba, State: Running, Role: LEADER
I20260923 15:15:17.251793 6923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46847 } }
I20260923 15:15:17.251849 6918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:17.252110 6923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46847 } } }
I20260923 15:15:17.252281 6923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:17.252480 6923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fabd4fbd7384405b4657f3eb4b066ba. Latest consensus state: current_term: 1 leader_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fabd4fbd7384405b4657f3eb4b066ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 46847 } } }
I20260923 15:15:17.252552 6923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:17.252923 6932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:17.253612 6932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:17.257256 6932 catalog_manager.cc:1347] Generated new cluster ID: a077086f5d234106b3571885a0ce72dc
I20260923 15:15:17.257311 6932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:17.261335 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.249523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.21.204.126:46847)
0923 15:15:17.249704 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.249711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.249735 (+ 24us) server_negotiation.cc:407] Connection header received
0923 15:15:17.249879 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.249902 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.249972 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.250127 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:17.250941 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.252113 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.253250 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.253824 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.254266 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.256872 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:17.256876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:17.256876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:17.256886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:17.259368 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.259774 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.259777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.259778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.259897 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.260111 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.260112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.260113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.260730 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:17.260919 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.261011 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.261138 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260923 15:15:17.278007 6932 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:17.278544 6932 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:17.284284 6932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba: Generated new TSK 0
I20260923 15:15:17.284461 6932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:15:17.312810 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46847
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:17.394138 6945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:17.394326 6945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:17.394348 6945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:17.395454 6945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:17.395494 6945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:17.395522 6945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:15:17.396970 6945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:17.397202 6945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:17.397425 6945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:17.402464 6950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:17.402478 6951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:17.402449 6953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:17.402571 6945 server_base.cc:1034] running on GCE node
I20260923 15:15:17.402936 6945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:17.403167 6945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:17.404289 6945 hybrid_clock.cc:648] HybridClock initialized: now 1790176517404277 us; error 28 us; skew 500 ppm
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:17.407178 6945 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:15:17.407579 6945 webserver.cc:460] Webserver started at http://127.21.204.65:41789/ using document root <none> and password file <none>
I20260923 15:15:17.407754 6945 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:17.407792 6945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:17.407881 6945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:17.408845 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "f197e83b0c4e38174f230d5c69396904"
server_key_iv: "c38b07190562b740ba51e9ab877b1c97"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.409142 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "f197e83b0c4e38174f230d5c69396904"
server_key_iv: "c38b07190562b740ba51e9ab877b1c97"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.410483 6945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:17.411218 6960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.411442 6945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:15:17.411541 6945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "f197e83b0c4e38174f230d5c69396904"
server_key_iv: "c38b07190562b740ba51e9ab877b1c97"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.411615 6945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:17.453138 6945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:17.455238 6945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:17.455399 6945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:17.455651 6945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:17.455962 6945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:17.455997 6945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.456020 6945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:17.456063 6945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.461758 6945 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:38421
I20260923 15:15:17.461862 7074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:38421 every 8 connection(s)
I20260923 15:15:17.462221 6945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:15:17.467751 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 6945
I20260923 15:15:17.467862 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:15:17.468057 22321 external_mini_cluster.cc:1428] Setting key dbbdc2112664123d650927764313432e
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:17.472349 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.462938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:47199 (local address 127.21.204.126:46847)
0923 15:15:17.462988 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.462990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.463206 (+ 216us) server_negotiation.cc:407] Connection header received
0923 15:15:17.463724 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.463726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.463738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.463793 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:17.464726 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.465238 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.466267 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.466488 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.466575 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.468979 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:17.468980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:17.468981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:17.468992 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:17.470739 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.471249 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.471250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.471250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.471302 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.471616 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.471617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.471617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.471767 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:17.471836 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.472094 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.472189 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:15:17.472357 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.462903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:46847 (local address 127.21.204.65:47199)
0923 15:15:17.463090 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:17.463124 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:17.463621 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:17.463860 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:17.463864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:17.464439 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:17.464622 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.464626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.465345 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.465347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.466173 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.466175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.466588 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.466589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.466714 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.467190 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:17.467198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:17.468904 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:17.470830 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:17.470837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:17.470840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:17.471166 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:17.471383 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:17.471384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:17.471385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:17.471541 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:17.471833 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:17.471839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:17.471956 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 15:15:17.472043 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 15:15:17.472164 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260923 15:15:17.472965 7075 heartbeater.cc:347] Connected to a master server at 127.21.204.126:46847
I20260923 15:15:17.473079 7075 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:17.473294 7075 heartbeater.cc:511] Master 127.21.204.126:46847 requested a full tablet report, sending...
I20260923 15:15:17.473956 6882 ts_manager.cc:194] Registered new tserver with Master: c6c79847ad1b4d44a457b9d2fb3c64e7 (127.21.204.65:38421)
I20260923 15:15:17.474963 6882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:47199
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:15:17.494920 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46847
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:17.577167 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:17.577406 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:17.577446 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:17.578621 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:17.578692 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:17.578749 7080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:15:17.580230 7080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:17.580493 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:17.580734 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:15:17.586056 7080 server_base.cc:1034] running on GCE node
W20260923 15:15:17.586022 7085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:17.586022 7088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:17.586031 7086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:17.586481 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:17.586778 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:17.587939 7080 hybrid_clock.cc:648] HybridClock initialized: now 1790176517587933 us; error 31 us; skew 500 ppm
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:17.590378 7080 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:15:17.590811 7080 webserver.cc:460] Webserver started at http://127.21.204.66:32909/ using document root <none> and password file <none>
I20260923 15:15:17.591048 7080 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:17.591116 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:17.591250 7080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:17.592276 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "6433543722b744d0a6aaba818ed8562a"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "4c98d733b5edb9a1e5ff1ff7d2db7aac"
server_key_iv: "f132432b451a49f70aae26c97b5306c5"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.592638 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "6433543722b744d0a6aaba818ed8562a"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "4c98d733b5edb9a1e5ff1ff7d2db7aac"
server_key_iv: "f132432b451a49f70aae26c97b5306c5"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.594085 7080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:15:17.594866 7095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.595072 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:15:17.595171 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "6433543722b744d0a6aaba818ed8562a"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "4c98d733b5edb9a1e5ff1ff7d2db7aac"
server_key_iv: "f132432b451a49f70aae26c97b5306c5"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.595254 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:17.604254 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:17.606077 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:17.606186 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:17.606338 7080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:17.606647 7080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:17.606703 7080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.606745 7080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:17.606774 7080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.612689 7080 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:44627
I20260923 15:15:17.612746 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:44627 every 8 connection(s)
I20260923 15:15:17.613030 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:17.619474 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 7080
I20260923 15:15:17.619588 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:15:17.619791 22321 external_mini_cluster.cc:1428] Setting key 66b2fd199fc7938bcfd535ddf8f15086
I20260923 15:15:17.622335 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.613728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:42419 (local address 127.21.204.126:46847)
0923 15:15:17.613808 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.613810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.614067 (+ 257us) server_negotiation.cc:407] Connection header received
0923 15:15:17.614596 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.614597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.614609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.614673 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:17.615596 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.616094 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.617016 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.617201 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.617252 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.619236 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:17.619237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:17.619238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:17.619243 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:17.620779 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.621256 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.621257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.621257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.621305 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.621578 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.621579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.621579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.621675 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:17.621723 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.622119 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.622213 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:15:17.622434 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.613711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:46847 (local address 127.21.204.66:42419)
0923 15:15:17.613929 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:17.613976 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:17.614505 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:17.614722 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:17.614725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:17.615262 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:17.615510 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.615515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.616174 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.616174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.616942 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.616944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.617274 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.617274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.617367 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.617782 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:17.617790 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:17.619154 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:17.620855 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:17.620862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:17.620864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:17.621181 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:17.621369 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:17.621370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:17.621370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:17.621515 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:17.621738 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:17.621744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:17.621950 (+ 206us) client_negotiation.cc:769] Sending connection context
0923 15:15:17.622083 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 15:15:17.622250 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260923 15:15:17.623100 7209 heartbeater.cc:347] Connected to a master server at 127.21.204.126:46847
I20260923 15:15:17.623214 7209 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:17.623420 7209 heartbeater.cc:511] Master 127.21.204.126:46847 requested a full tablet report, sending...
I20260923 15:15:17.623780 6882 ts_manager.cc:194] Registered new tserver with Master: 6433543722b744d0a6aaba818ed8562a (127.21.204.66:44627)
I20260923 15:15:17.624152 6882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:42419
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:15:17.644542 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46847
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:32795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:17.725704 7214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:17.725987 7214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:17.726029 7214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:17.727254 7214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:17.727325 7214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:17.727376 7214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:15:17.728871 7214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:32795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:46847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:17.729142 7214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:17.729377 7214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:17.734469 7222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:17.734496 7219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:17.734479 7220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:17.734525 7214 server_base.cc:1034] running on GCE node
I20260923 15:15:17.734946 7214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:17.735185 7214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:17.736336 7214 hybrid_clock.cc:648] HybridClock initialized: now 1790176517736317 us; error 32 us; skew 500 ppm
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:17.738785 7214 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:15:17.739149 7214 webserver.cc:460] Webserver started at http://127.21.204.67:42229/ using document root <none> and password file <none>
I20260923 15:15:17.739346 7214 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:17.739387 7214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:17.739475 7214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:17.740512 7214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "ea4102d88cb141f2a44b9c0a625fea5a"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "cece14be5dcfeaca0e5fcf73dfc71a98"
server_key_iv: "3ea01658f78905f74b71b5a7acc44b7c"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.740813 7214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "ea4102d88cb141f2a44b9c0a625fea5a"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "cece14be5dcfeaca0e5fcf73dfc71a98"
server_key_iv: "3ea01658f78905f74b71b5a7acc44b7c"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.742137 7214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:17.742990 7229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.743162 7214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:17.743256 7214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "ea4102d88cb141f2a44b9c0a625fea5a"
format_stamp: "Formatted at 2026-09-23 15:15:17 on dist-test-slave-t5lt"
server_key: "cece14be5dcfeaca0e5fcf73dfc71a98"
server_key_iv: "3ea01658f78905f74b71b5a7acc44b7c"
server_key_version: "encryptionkey@0"
I20260923 15:15:17.743314 7214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:17.752367 7214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:17.754462 7214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:17.754576 7214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:17.754758 7214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:17.755003 7214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:17.755035 7214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.755057 7214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:17.755071 7214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:17.761356 7214 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:34485
I20260923 15:15:17.761397 7342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:34485 every 8 connection(s)
I20260923 15:15:17.761775 7214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:15:17 dist-test-slave-t5lt krb5kdc[6836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:17.769079 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 7214
I20260923 15:15:17.769183 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:15:17.769373 22321 external_mini_cluster.cc:1428] Setting key e4e43e9477e5c0e02475e559f5ed30b2
I20260923 15:15:17.770809 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.762572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:59695 (local address 127.21.204.126:46847)
0923 15:15:17.762646 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.762648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.762875 (+ 227us) server_negotiation.cc:407] Connection header received
0923 15:15:17.763333 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.763334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.763345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.763408 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:17.764297 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.764785 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.765774 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.766018 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.766074 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.767980 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:17.767981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:17.767981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:17.767988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:17.769385 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.769848 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.769849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.769850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.769977 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.770234 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.770235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.770236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.770382 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:17.770468 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.770677 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.770739 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 15:15:17.771008 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.762555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:46847 (local address 127.21.204.67:59695)
0923 15:15:17.762720 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:17.762761 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:17.763210 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:17.763459 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:17.763463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:17.763998 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:17.764177 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.764181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.764888 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.764890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.765694 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.765696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.766084 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.766085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.766195 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.766611 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:17.766618 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:17.767885 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:17.769483 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:17.769492 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:17.769494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:17.769755 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:17.770048 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:17.770049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:17.770050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:17.770170 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:17.770466 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:17.770472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:17.770580 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 15:15:17.770660 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 15:15:17.770776 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260923 15:15:17.771587 7343 heartbeater.cc:347] Connected to a master server at 127.21.204.126:46847
I20260923 15:15:17.771692 7343 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:17.771862 7343 heartbeater.cc:511] Master 127.21.204.126:46847 requested a full tablet report, sending...
I20260923 15:15:17.772253 6882 ts_manager.cc:194] Registered new tserver with Master: ea4102d88cb141f2a44b9c0a625fea5a (127.21.204.67:34485)
I20260923 15:15:17.772620 6882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:59695
I20260923 15:15:17.781327 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:15:17.788724 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.782743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.21.204.126:46847)
0923 15:15:17.782790 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.782792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.782886 (+ 94us) server_negotiation.cc:407] Connection header received
0923 15:15:17.782923 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.782924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.782932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.782978 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:17.783534 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.784011 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.784816 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.784998 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.785056 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.786155 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:17.786156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:17.786156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:17.786162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:17.787363 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.787761 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.787762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.787763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.787819 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:17.788041 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:17.788042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:17.788043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:17.788228 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:17.788359 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.788517 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.788591 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:15:17.790208 6882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:15:17.791802 6882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:15:17.798216 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.794158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:38421 (local address 127.0.0.1:50520)
0923 15:15:17.794581 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:17.794590 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:17.794643 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:17.794937 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:17.794940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:17.794943 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:17.795019 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.795021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.795930 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.795932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.797053 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.797056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.797923 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.797925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.798008 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.798022 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 15:15:17.798099 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 15:15:17.798145 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":75,"threads_started":1}
I20260923 15:15:17.798264 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.794424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.21.204.65:38421)
0923 15:15:17.794584 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.794589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.794640 (+ 51us) server_negotiation.cc:407] Connection header received
0923 15:15:17.794714 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.794716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.794729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.794836 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:17.795110 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.795824 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.797127 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.797785 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.797865 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.797972 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.798126 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.798170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260923 15:15:17.798718 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.794152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:34485 (local address 127.0.0.1:37578)
0923 15:15:17.794367 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:17.794406 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:17.794471 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:17.795009 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:17.795010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:17.795012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:17.795083 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.795085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.796408 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.796409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.797361 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.797363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.798473 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.798474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.798574 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.798578 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:17.798614 (+ 36us) client_negotiation.cc:240] Negotiation successful
0923 15:15:17.798652 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260923 15:15:17.799134 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.794230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.21.204.67:34485)
0923 15:15:17.794782 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.794785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.794852 (+ 67us) server_negotiation.cc:407] Connection header received
0923 15:15:17.794894 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.794896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.794913 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.795163 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:17.795235 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.796279 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.797440 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.798216 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.798892 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.798955 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.799003 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.799045 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":75,"threads_started":1}
I20260923 15:15:17.799037 7009 tablet_service.cc:1467] Processing CreateTablet for tablet afa3f14b8adf4b93ad78363627fb128a (DEFAULT_TABLE table=test-table [id=2c2a0cf9e20442419e39d5646adc00c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:17.799431 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa3f14b8adf4b93ad78363627fb128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:17.799868 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.794197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:44627 (local address 127.0.0.1:55332)
0923 15:15:17.794694 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:17.794703 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:17.794760 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:17.795015 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:17.795015 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:17.795017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:17.795074 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.795076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.797430 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.797431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.798471 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:17.798475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.799643 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.799644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:17.799705 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.799709 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:17.799765 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 15:15:17.799803 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":7,"thread_start_us":104,"threads_started":1}
I20260923 15:15:17.799824 7277 tablet_service.cc:1467] Processing CreateTablet for tablet afa3f14b8adf4b93ad78363627fb128a (DEFAULT_TABLE table=test-table [id=2c2a0cf9e20442419e39d5646adc00c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:17.799899 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:17.794450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.21.204.66:44627)
0923 15:15:17.794654 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:17.794658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:17.794785 (+ 127us) server_negotiation.cc:407] Connection header received
0923 15:15:17.794840 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:17.794842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:17.794859 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:17.794933 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:17.795140 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.796300 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.798553 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:17.799559 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:17.799631 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:17.799683 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:17.799777 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 15:15:17.799821 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260923 15:15:17.800128 7277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa3f14b8adf4b93ad78363627fb128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:17.800555 7143 tablet_service.cc:1467] Processing CreateTablet for tablet afa3f14b8adf4b93ad78363627fb128a (DEFAULT_TABLE table=test-table [id=2c2a0cf9e20442419e39d5646adc00c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:17.800891 7143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa3f14b8adf4b93ad78363627fb128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:17.802939 7360 tablet_bootstrap.cc:492] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: Bootstrap starting.
I20260923 15:15:17.803361 7361 tablet_bootstrap.cc:492] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Bootstrap starting.
I20260923 15:15:17.803610 7360 tablet_bootstrap.cc:654] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:17.804128 7361 tablet_bootstrap.cc:654] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:17.804226 7360 log.cc:824] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:17.804454 7362 tablet_bootstrap.cc:492] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Bootstrap starting.
I20260923 15:15:17.804746 7361 log.cc:824] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:17.805102 7360 tablet_bootstrap.cc:492] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: No bootstrap required, opened a new log
I20260923 15:15:17.805183 7360 ts_tablet_manager.cc:1397] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:17.805217 7362 tablet_bootstrap.cc:654] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:17.805516 7361 tablet_bootstrap.cc:492] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: No bootstrap required, opened a new log
I20260923 15:15:17.805593 7361 ts_tablet_manager.cc:1397] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:17.805768 7362 log.cc:824] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:17.806481 7362 tablet_bootstrap.cc:492] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: No bootstrap required, opened a new log
I20260923 15:15:17.806564 7362 ts_tablet_manager.cc:1397] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:17.807524 7361 raft_consensus.cc:348] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:17.807524 7360 raft_consensus.cc:348] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:17.807706 7361 raft_consensus.cc:374] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:17.807706 7360 raft_consensus.cc:374] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:17.807744 7360 raft_consensus.cc:729] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c79847ad1b4d44a457b9d2fb3c64e7, State: Initialized, Role: FOLLOWER
I20260923 15:15:17.807744 7361 raft_consensus.cc:729] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4102d88cb141f2a44b9c0a625fea5a, State: Initialized, Role: FOLLOWER
I20260923 15:15:17.807889 7360 consensus_queue.cc:260] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:17.807889 7361 consensus_queue.cc:260] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:17.808018 7362 raft_consensus.cc:348] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:17.808192 7360 ts_tablet_manager.cc:1428] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:15:17.808192 7361 ts_tablet_manager.cc:1428] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:15:17.808200 7362 raft_consensus.cc:374] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:17.808245 7362 raft_consensus.cc:729] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6433543722b744d0a6aaba818ed8562a, State: Initialized, Role: FOLLOWER
I20260923 15:15:17.808195 7343 heartbeater.cc:503] Master 127.21.204.126:46847 was elected leader, sending a full tablet report...
I20260923 15:15:17.808329 7075 heartbeater.cc:503] Master 127.21.204.126:46847 was elected leader, sending a full tablet report...
I20260923 15:15:17.808375 7362 consensus_queue.cc:260] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:17.808641 7362 ts_tablet_manager.cc:1428] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:17.808771 7209 heartbeater.cc:503] Master 127.21.204.126:46847 was elected leader, sending a full tablet report...
W20260923 15:15:17.863673 7210 tablet.cc:2369] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:15:17.962954 7076 tablet.cc:2369] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:15:18.012485 7344 tablet.cc:2369] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:15:18.238451 7366 raft_consensus.cc:482] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:15:18.238576 7366 raft_consensus.cc:504] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:18.238989 7366 leader_election.cc:288] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea4102d88cb141f2a44b9c0a625fea5a (127.21.204.67:34485), 6433543722b744d0a6aaba818ed8562a (127.21.204.66:44627)
I20260923 15:15:18.243139 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.239308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:36777 (local address 127.21.204.67:34485)
0923 15:15:18.239353 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.239355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.239364 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:18.239445 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.239446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.239456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.239524 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:18.239747 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.240509 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.242161 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.242735 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.242811 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.242821 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.243039 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.243076 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:15:18.243144 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.239263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:51843 (local address 127.21.204.66:44627)
0923 15:15:18.239326 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.239328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.239369 (+ 41us) server_negotiation.cc:407] Connection header received
0923 15:15:18.239444 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.239445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.239458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.239524 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:18.239748 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.240510 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.242161 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.242735 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.242811 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.242821 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.243030 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.243078 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 15:15:18.243158 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.239175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:34485 (local address 127.21.204.65:36777)
0923 15:15:18.239305 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.239318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.239372 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.239547 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.239548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.239551 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:18.239673 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.239674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.240601 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.240602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.242070 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.242074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.242874 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.242876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.242956 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.242961 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.243019 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.243064 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":95,"thread_start_us":63,"threads_started":1}
I20260923 15:15:18.243400 7369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.239175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:44627 (local address 127.21.204.65:51843)
0923 15:15:18.239305 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.239318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.239373 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.239547 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.239548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.239552 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:18.239665 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.239667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.240601 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.240602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.242070 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.242074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.242828 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.242829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.242925 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.242929 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.243263 (+ 334us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.243312 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":122,"thread_start_us":63,"threads_started":1}
I20260923 15:15:18.243592 7297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3f14b8adf4b93ad78363627fb128a" candidate_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" is_pre_election: true
I20260923 15:15:18.243749 7297 raft_consensus.cc:2454] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c79847ad1b4d44a457b9d2fb3c64e7 in term 0.
I20260923 15:15:18.243769 7163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3f14b8adf4b93ad78363627fb128a" candidate_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6433543722b744d0a6aaba818ed8562a" is_pre_election: true
I20260923 15:15:18.243898 7163 raft_consensus.cc:2454] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c79847ad1b4d44a457b9d2fb3c64e7 in term 0.
I20260923 15:15:18.244020 6962 leader_election.cc:302] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c79847ad1b4d44a457b9d2fb3c64e7, ea4102d88cb141f2a44b9c0a625fea5a; no voters:
I20260923 15:15:18.244189 7366 raft_consensus.cc:2781] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:15:18.244261 7366 raft_consensus.cc:482] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:15:18.244287 7366 raft_consensus.cc:3037] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:18.244999 7366 raft_consensus.cc:504] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:18.245144 7366 leader_election.cc:288] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [CANDIDATE]: Term 1 election: Requested vote from peers ea4102d88cb141f2a44b9c0a625fea5a (127.21.204.67:34485), 6433543722b744d0a6aaba818ed8562a (127.21.204.66:44627)
I20260923 15:15:18.245322 7163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3f14b8adf4b93ad78363627fb128a" candidate_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6433543722b744d0a6aaba818ed8562a"
I20260923 15:15:18.245345 7297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa3f14b8adf4b93ad78363627fb128a" candidate_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea4102d88cb141f2a44b9c0a625fea5a"
I20260923 15:15:18.245412 7163 raft_consensus.cc:3037] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:18.245412 7297 raft_consensus.cc:3037] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:18.246343 7163 raft_consensus.cc:2454] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c79847ad1b4d44a457b9d2fb3c64e7 in term 1.
I20260923 15:15:18.246345 7297 raft_consensus.cc:2454] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c79847ad1b4d44a457b9d2fb3c64e7 in term 1.
I20260923 15:15:18.246510 6962 leader_election.cc:302] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c79847ad1b4d44a457b9d2fb3c64e7, ea4102d88cb141f2a44b9c0a625fea5a; no voters:
I20260923 15:15:18.246637 7366 raft_consensus.cc:2781] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:18.246824 7366 raft_consensus.cc:686] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [term 1 LEADER]: Becoming Leader. State: Replica: c6c79847ad1b4d44a457b9d2fb3c64e7, State: Running, Role: LEADER
I20260923 15:15:18.246968 7366 consensus_queue.cc:237] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } }
I20260923 15:15:18.247795 6882 catalog_manager.cc:5495] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c79847ad1b4d44a457b9d2fb3c64e7 (127.21.204.65). New cstate: current_term: 1 leader_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c79847ad1b4d44a457b9d2fb3c64e7" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 38421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 } health_report { overall_health: UNKNOWN } } }
I20260923 15:15:18.347563 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.344836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.21.204.65:38421)
0923 15:15:18.344956 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.344959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.345035 (+ 76us) server_negotiation.cc:407] Connection header received
0923 15:15:18.345147 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.345148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.345168 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.345256 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:18.345417 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.345951 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.346729 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.346933 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.347044 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.347215 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:18.347321 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:18.347425 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.347443 (+ 18us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.347462 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260923 15:15:18.349265 7297 raft_consensus.cc:1261] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 1 FOLLOWER]: Refusing update from remote peer c6c79847ad1b4d44a457b9d2fb3c64e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:18.349308 7163 raft_consensus.cc:1261] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 1 FOLLOWER]: Refusing update from remote peer c6c79847ad1b4d44a457b9d2fb3c64e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:18.349628 7371 consensus_queue.cc:1035] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea4102d88cb141f2a44b9c0a625fea5a" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:18.349768 7366 consensus_queue.cc:1035] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6433543722b744d0a6aaba818ed8562a" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:18.352254 7376 mvcc.cc:205] Tried to move back new op lower bound from 7332563019155849216 to 7332563018740215808. Current Snapshot: MvccSnapshot[applied={T|T < 7332563019155849216}]
I20260923 15:15:18.352942 7380 mvcc.cc:205] Tried to move back new op lower bound from 7332563019155849216 to 7332563018740215808. Current Snapshot: MvccSnapshot[applied={T|T < 7332563019155849216}]
I20260923 15:15:18.355142 6882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:18.355258 6882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:18.355996 6882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fabd4fbd7384405b4657f3eb4b066ba: Sending DeleteTablet for 3 replicas of tablet afa3f14b8adf4b93ad78363627fb128a
I20260923 15:15:18.356319 7277 tablet_service.cc:1514] Processing DeleteTablet for tablet afa3f14b8adf4b93ad78363627fb128a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:18 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:37578
I20260923 15:15:18.356318 7009 tablet_service.cc:1514] Processing DeleteTablet for tablet afa3f14b8adf4b93ad78363627fb128a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:18 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:50520
I20260923 15:15:18.356323 7143 tablet_service.cc:1514] Processing DeleteTablet for tablet afa3f14b8adf4b93ad78363627fb128a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:18 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:55332
I20260923 15:15:18.356518 7388 tablet_replica.cc:330] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: stopping tablet replica
I20260923 15:15:18.356709 7388 raft_consensus.cc:2229] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:18.356806 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 6945
I20260923 15:15:18.356709 7387 tablet_replica.cc:330] T afa3f14b8adf4b93ad78363627fb128a P c6c79847ad1b4d44a457b9d2fb3c64e7: stopping tablet replica
I20260923 15:15:18.357671 7388 raft_consensus.cc:2258] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:18.358045 7388 ts_tablet_manager.cc:1905] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:18.359212 7389 tablet_replica.cc:330] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: stopping tablet replica
I20260923 15:15:18.359332 7389 raft_consensus.cc:2229] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:18.359438 7389 raft_consensus.cc:2258] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:18.359527 7388 ts_tablet_manager.cc:1918] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:18.359597 7388 log.cc:1196] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/afa3f14b8adf4b93ad78363627fb128a
I20260923 15:15:18.359761 7389 ts_tablet_manager.cc:1905] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:18.359838 7388 ts_tablet_manager.cc:1939] T afa3f14b8adf4b93ad78363627fb128a P ea4102d88cb141f2a44b9c0a625fea5a: Deleting consensus metadata
I20260923 15:15:18.360249 6867 catalog_manager.cc:4841] TS ea4102d88cb141f2a44b9c0a625fea5a (127.21.204.67:34485): tablet afa3f14b8adf4b93ad78363627fb128a (table test-table [id=2c2a0cf9e20442419e39d5646adc00c8]) successfully deleted
I20260923 15:15:18.361001 7389 ts_tablet_manager.cc:1918] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:18.361057 7389 log.cc:1196] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/afa3f14b8adf4b93ad78363627fb128a
I20260923 15:15:18.361289 7389 ts_tablet_manager.cc:1939] T afa3f14b8adf4b93ad78363627fb128a P 6433543722b744d0a6aaba818ed8562a: Deleting consensus metadata
I20260923 15:15:18.361845 6869 catalog_manager.cc:4841] TS 6433543722b744d0a6aaba818ed8562a (127.21.204.66:44627): tablet afa3f14b8adf4b93ad78363627fb128a (table test-table [id=2c2a0cf9e20442419e39d5646adc00c8]) successfully deleted
W20260923 15:15:18.362665 6868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:38421 (error 108)
I20260923 15:15:18.363195 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 7080
I20260923 15:15:18.363217 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.363030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:38421 (local address 127.0.0.1:50550)
0923 15:15:18.363133 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.363179 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:38421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":39,"threads_started":1}
W20260923 15:15:18.363996 6868 catalog_manager.cc:4584] TS c6c79847ad1b4d44a457b9d2fb3c64e7 (127.21.204.65:38421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afa3f14b8adf4b93ad78363627fb128a: Network error: Client connection negotiation failed: client connection to 127.21.204.65:38421: connect: Connection refused (error 111)
I20260923 15:15:18.368247 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 7214
I20260923 15:15:18.372632 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 6850
2026-09-23T15:15:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2409 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T15:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:18Z Disabled control of system clock
I20260923 15:15:18.399780 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34335
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:18.478847 7395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:18.479051 7395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:18.479072 7395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:18.480123 7395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:18.480157 7395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:18.480170 7395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:18.480182 7395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:18.480194 7395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:18.481442 7395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:34335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:18.481626 7395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:18.481848 7395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:18.486639 7400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:18.486632 7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:18.486779 7395 server_base.cc:1034] running on GCE node
W20260923 15:15:18.486667 7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:18.487092 7395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:18.487322 7395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:18.488458 7395 hybrid_clock.cc:648] HybridClock initialized: now 1790176518488435 us; error 46 us; skew 500 ppm
I20260923 15:15:18.489756 7395 webserver.cc:460] Webserver started at http://127.21.204.126:36423/ using document root <none> and password file <none>
I20260923 15:15:18.490062 7395 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:18.490115 7395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:18.490262 7395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:18.491273 7395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "fd8d381a8c4b427eaad621b26780e20d"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "fba74eff7eef1ecf9dedab86006ac6c6"
server_key_iv: "52047dd47c5b3e6c279844a0aff1f16c"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.491638 7395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "fd8d381a8c4b427eaad621b26780e20d"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "fba74eff7eef1ecf9dedab86006ac6c6"
server_key_iv: "52047dd47c5b3e6c279844a0aff1f16c"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.492944 7395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:18.493680 7409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.493974 7395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:18.494074 7395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "fd8d381a8c4b427eaad621b26780e20d"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "fba74eff7eef1ecf9dedab86006ac6c6"
server_key_iv: "52047dd47c5b3e6c279844a0aff1f16c"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.494161 7395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:18.503988 7395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:18.504276 7395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:18.504423 7395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:18.508648 7395 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:34335
I20260923 15:15:18.508679 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:34335 every 8 connection(s)
I20260923 15:15:18.509014 7395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:18.510010 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:18.512176 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d: Bootstrap starting.
I20260923 15:15:18.512696 7462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:18.512929 7462 log.cc:824] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:18.513666 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d: No bootstrap required, opened a new log
I20260923 15:15:18.513701 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 7395
I20260923 15:15:18.513814 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:18.514050 22321 external_mini_cluster.cc:1428] Setting key d18d64d554c534e5b7c781ac2a40ecec
I20260923 15:15:18.515688 7462 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d381a8c4b427eaad621b26780e20d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34335 } }
I20260923 15:15:18.515851 7462 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:18.515890 7462 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8d381a8c4b427eaad621b26780e20d, State: Initialized, Role: FOLLOWER
I20260923 15:15:18.516021 7462 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d381a8c4b427eaad621b26780e20d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34335 } }
I20260923 15:15:18.516119 7462 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:18.516170 7462 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:18.516218 7462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:18.516722 7462 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d381a8c4b427eaad621b26780e20d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34335 } }
I20260923 15:15:18.516805 7462 leader_election.cc:302] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8d381a8c4b427eaad621b26780e20d; no voters:
I20260923 15:15:18.517037 7462 leader_election.cc:288] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:18.517063 7467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:18.517174 7467 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [term 1 LEADER]: Becoming Leader. State: Replica: fd8d381a8c4b427eaad621b26780e20d, State: Running, Role: LEADER
I20260923 15:15:18.517257 7467 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d381a8c4b427eaad621b26780e20d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34335 } }
I20260923 15:15:18.517303 7462 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:18.517586 7467 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8d381a8c4b427eaad621b26780e20d. Latest consensus state: current_term: 1 leader_uuid: "fd8d381a8c4b427eaad621b26780e20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d381a8c4b427eaad621b26780e20d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34335 } } }
I20260923 15:15:18.517652 7467 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:18.517618 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8d381a8c4b427eaad621b26780e20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8d381a8c4b427eaad621b26780e20d" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 34335 } } }
I20260923 15:15:18.517702 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:18.518019 7471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:18.518546 7471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:18.520442 7471 catalog_manager.cc:1347] Generated new cluster ID: ed8e915238434da591c76acc41a368e1
I20260923 15:15:18.520501 7471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:18.520947 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.514538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.21.204.126:34335)
0923 15:15:18.514742 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.514749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.514772 (+ 23us) server_negotiation.cc:407] Connection header received
0923 15:15:18.515671 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.515678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.515731 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.515836 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:18.516081 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.517640 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.518803 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.519230 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.519360 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.520097 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:18.520101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:18.520101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:18.520362 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:18.520416 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:18.520432 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:18.520491 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.520584 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.520758 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260923 15:15:18.536342 7471 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:18.537009 7471 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:18.544184 7471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d: Generated new TSK 0
I20260923 15:15:18.544366 7471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:18.554075 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34335
--builtin_ntp_servers=127.21.204.84:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:18.634193 7486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:18.634430 7486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:18.634467 7486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:18.635617 7486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:18.635679 7486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:18.635723 7486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:15:18.637087 7486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:18.637344 7486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:18.637647 7486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:18.643611 7491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:18.643658 7492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:18.643788 7486 server_base.cc:1034] running on GCE node
W20260923 15:15:18.643936 7494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:18.644198 7486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:18.644507 7486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:18.645700 7486 hybrid_clock.cc:648] HybridClock initialized: now 1790176518645688 us; error 27 us; skew 500 ppm
I20260923 15:15:18.647084 7486 webserver.cc:460] Webserver started at http://127.21.204.65:33827/ using document root <none> and password file <none>
I20260923 15:15:18.647329 7486 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:18.647385 7486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:18.647500 7486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:18.648872 7486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "b3bd97f0557d4953b75e190e64bbd2dc"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "19db8365e3867e6049f3c3c9b01ac2f6"
server_key_iv: "dbd95b3cf24b2d6557005de8b51d62c2"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.649250 7486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "b3bd97f0557d4953b75e190e64bbd2dc"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "19db8365e3867e6049f3c3c9b01ac2f6"
server_key_iv: "dbd95b3cf24b2d6557005de8b51d62c2"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.650905 7486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260923 15:15:18.651826 7500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.652035 7486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:18.652128 7486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "b3bd97f0557d4953b75e190e64bbd2dc"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "19db8365e3867e6049f3c3c9b01ac2f6"
server_key_iv: "dbd95b3cf24b2d6557005de8b51d62c2"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.652195 7486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:18.697060 7486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:18.697366 7486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:18.697474 7486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:18.697736 7486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:18.698252 7486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:18.698299 7486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.698325 7486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:18.698398 7486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.704221 7486 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:42649
I20260923 15:15:18.704285 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:42649 every 8 connection(s)
I20260923 15:15:18.704614 7486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:15:18.709406 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 7486
I20260923 15:15:18.709532 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:15:18.709715 22321 external_mini_cluster.cc:1428] Setting key 33f1a94fc9ac544a63d9e9e39a30e8dc
I20260923 15:15:18.710178 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.705357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:57357 (local address 127.21.204.126:34335)
0923 15:15:18.705438 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.705440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.705760 (+ 320us) server_negotiation.cc:407] Connection header received
0923 15:15:18.706292 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.706293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.706307 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.706372 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:18.707110 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.707614 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.708564 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.708850 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.708908 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.709774 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:18.709776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:18.709776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:18.709785 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:18.709804 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:18.709809 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:18.709934 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.710096 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.710118 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 15:15:18.710357 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.705340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34335 (local address 127.21.204.65:57357)
0923 15:15:18.705586 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.705631 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.706180 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.706455 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.706459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.706470 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:18.707026 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.707030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.707695 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.707696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.708461 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.708463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.708929 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.708930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.709036 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.709625 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:18.709626 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:18.709668 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:18.709691 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:18.709939 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:18.709946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:18.709950 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.710039 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.710141 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260923 15:15:18.710903 7614 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34335
I20260923 15:15:18.710939 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34335
--builtin_ntp_servers=127.21.204.84:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 15:15:18.711009 7614 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:18.711194 7614 heartbeater.cc:511] Master 127.21.204.126:34335 requested a full tablet report, sending...
I20260923 15:15:18.711687 7426 ts_manager.cc:194] Registered new tserver with Master: b3bd97f0557d4953b75e190e64bbd2dc (127.21.204.65:42649)
I20260923 15:15:18.712306 7426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.65:57357
W20260923 15:15:18.791111 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:18.791347 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:18.791385 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:18.792510 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:18.792577 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:18.792618 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:15:18.793974 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:18.794245 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:18.794544 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:18.799520 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:18.799563 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:18.799638 7617 server_base.cc:1034] running on GCE node
W20260923 15:15:18.799556 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:18.799932 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:18.800176 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:18.801342 7617 hybrid_clock.cc:648] HybridClock initialized: now 1790176518801332 us; error 44 us; skew 500 ppm
I20260923 15:15:18.802691 7617 webserver.cc:460] Webserver started at http://127.21.204.66:37263/ using document root <none> and password file <none>
I20260923 15:15:18.802942 7617 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:18.803012 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:18.803161 7617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:18.804208 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "7ffd76120938c1e30e6b079702f0b586"
server_key_iv: "6737334046200b1a1b8f3c665e4fa8ac"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.804559 7617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "7ffd76120938c1e30e6b079702f0b586"
server_key_iv: "6737334046200b1a1b8f3c665e4fa8ac"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.806008 7617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:18.806824 7631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.807017 7617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:18.807114 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "7ffd76120938c1e30e6b079702f0b586"
server_key_iv: "6737334046200b1a1b8f3c665e4fa8ac"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.807200 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:18.822402 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:18.822711 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:18.822865 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:18.823102 7617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:18.823437 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:18.823493 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.823539 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:18.823561 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.829399 7617 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:40207
I20260923 15:15:18.829490 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:40207 every 8 connection(s)
I20260923 15:15:18.829797 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:15:18.834981 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.830613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:40727 (local address 127.21.204.126:34335)
0923 15:15:18.830661 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.830663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.830884 (+ 221us) server_negotiation.cc:407] Connection header received
0923 15:15:18.831376 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.831378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.831397 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.831453 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:18.832095 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.832627 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.833625 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.833823 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.833902 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.834734 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:18.834736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:18.834736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:18.834740 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:18.834755 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:18.834759 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:18.834817 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.834900 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.834923 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:18.835161 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.830596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34335 (local address 127.21.204.66:40727)
0923 15:15:18.830767 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.830799 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.831261 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.831510 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.831514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.831521 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:18.832019 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.832023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.832741 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.832742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.833540 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.833542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.833953 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.833954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.834089 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.834590 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:18.834591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:18.834618 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:18.834639 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:18.834827 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:18.834832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:18.834834 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.834887 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.834968 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260923 15:15:18.835654 7745 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34335
I20260923 15:15:18.835754 7745 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:18.835898 7745 heartbeater.cc:511] Master 127.21.204.126:34335 requested a full tablet report, sending...
I20260923 15:15:18.836236 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 7617
I20260923 15:15:18.836359 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:15:18.836354 7426 ts_manager.cc:194] Registered new tserver with Master: 12bb4d41c6e84b3a9241ad2ad91cb73d (127.21.204.66:40207)
I20260923 15:15:18.836565 22321 external_mini_cluster.cc:1428] Setting key 55d75c382312ebc924412dbd28da9fac
I20260923 15:15:18.836778 7426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.66:40727
I20260923 15:15:18.837450 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34335
--builtin_ntp_servers=127.21.204.84:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:18.921363 7748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:18.921674 7748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:18.921718 7748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:18.923785 7748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:18.923847 7748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:18.923892 7748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:15:18.926158 7748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:34335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:18.926440 7748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:18.926783 7748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:18.933625 7754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:18.933632 7756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:18.933632 7753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:18.934027 7748 server_base.cc:1034] running on GCE node
I20260923 15:15:18.934284 7748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:18.934598 7748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:18.935722 7748 hybrid_clock.cc:648] HybridClock initialized: now 1790176518935697 us; error 43 us; skew 500 ppm
I20260923 15:15:18.937388 7748 webserver.cc:460] Webserver started at http://127.21.204.67:34941/ using document root <none> and password file <none>
I20260923 15:15:18.937676 7748 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:18.937749 7748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:18.937939 7748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:18.939277 7748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "fcbf213c8a417120708800eb24b339e4"
server_key_iv: "6a4c9f1f190be6a9dec19e1c563c73e7"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.939706 7748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "fcbf213c8a417120708800eb24b339e4"
server_key_iv: "6a4c9f1f190be6a9dec19e1c563c73e7"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.941375 7748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:18.942410 7762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.942602 7748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:15:18.942705 7748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9"
format_stamp: "Formatted at 2026-09-23 15:15:18 on dist-test-slave-t5lt"
server_key: "fcbf213c8a417120708800eb24b339e4"
server_key_iv: "6a4c9f1f190be6a9dec19e1c563c73e7"
server_key_version: "encryptionkey@0"
I20260923 15:15:18.942793 7748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:18.960333 7748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:18.960661 7748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:18.960820 7748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:18.961054 7748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:18.961400 7748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:18.961457 7748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.961503 7748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:18.961530 7748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:18.967578 7748 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:45315
I20260923 15:15:18.967617 7877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:45315 every 8 connection(s)
I20260923 15:15:18.967973 7748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:15:18.972270 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 7748
I20260923 15:15:18.972411 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:15:18.972620 22321 external_mini_cluster.cc:1428] Setting key d6950b16a06b5b0a5aa22ac10e9913ce
I20260923 15:15:18.974165 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.968931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:47053 (local address 127.21.204.126:34335)
0923 15:15:18.968997 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.968999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.969247 (+ 248us) server_negotiation.cc:407] Connection header received
0923 15:15:18.969799 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.969800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.969814 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.969898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:18.970601 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.971115 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.972111 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.972315 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.972391 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.973617 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:18.973618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:18.973618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:18.973622 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:18.973637 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:18.973641 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:18.973817 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.974058 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.974092 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 15:15:18.974424 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.968931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:34335 (local address 127.21.204.67:47053)
0923 15:15:18.969122 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.969156 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.969689 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.969987 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.969991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.969999 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:18.970494 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.970498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.971216 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.971218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.971996 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.971998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.972419 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.972421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.972580 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.973386 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:18.973386 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:18.973416 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:18.973486 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:18.973927 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:18.973936 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:18.973939 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.974032 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.974140 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260923 15:15:18.974982 7878 heartbeater.cc:347] Connected to a master server at 127.21.204.126:34335
I20260923 15:15:18.975080 7878 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:18.975270 7878 heartbeater.cc:511] Master 127.21.204.126:34335 requested a full tablet report, sending...
I20260923 15:15:18.975708 7426 ts_manager.cc:194] Registered new tserver with Master: abcd79a2b89c48beb8c5677ab7e9b9e9 (127.21.204.67:45315)
I20260923 15:15:18.976105 7426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.67:47053
I20260923 15:15:18.985039 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:15:18.989669 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.986626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.21.204.126:34335)
0923 15:15:18.986688 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.986690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.986743 (+ 53us) server_negotiation.cc:407] Connection header received
0923 15:15:18.986796 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.986797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.986807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.986860 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:18.987143 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.987638 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.988458 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.988642 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.988705 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.989451 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:18.989452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:18.989453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:18.989457 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:18.989471 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:18.989475 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:18.989523 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.989585 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.989608 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 15:15:18.991372 7426 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:15:18.993052 7426 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:15:18.999656 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.995333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:45315 (local address 127.0.0.1:48516)
0923 15:15:18.995803 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.995814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.995881 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.996142 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.996144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.996147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:18.996213 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.996215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.997629 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.997630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.998716 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.998718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.999370 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.999371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.999446 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.999459 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.999535 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.999587 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":81,"threads_started":1}
I20260923 15:15:18.999668 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.995631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.21.204.67:45315)
0923 15:15:18.995791 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.995796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.995873 (+ 77us) server_negotiation.cc:407] Connection header received
0923 15:15:18.995957 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.995960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.995977 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.996058 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:18.996292 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.997545 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.998786 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.999267 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.999366 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.999469 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.999548 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.999585 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260923 15:15:18.999967 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.995632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.21.204.66:40207)
0923 15:15:18.995827 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.995833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.996143 (+ 310us) server_negotiation.cc:407] Connection header received
0923 15:15:18.996186 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.996188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.996204 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.996272 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:18.996459 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.997657 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.998757 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.999450 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.999526 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.999613 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:18.999833 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 15:15:18.999881 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260923 15:15:19.000140 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.995845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.21.204.65:42649)
0923 15:15:18.996065 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:18.996069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:18.996080 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:15:18.996122 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:18.996124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:18.996140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:18.996384 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:18.996452 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.997213 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.998716 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.999458 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.999885 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.999986 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:19.000013 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 15:15:19.000061 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
I20260923 15:15:18.999789 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.995326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:42649 (local address 127.0.0.1:59378)
0923 15:15:18.995462 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.995500 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.995585 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.996249 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.996250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.996252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:18.996306 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.996307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.997301 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.997301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.998622 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.998625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.999557 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.999558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.999620 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.999624 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.999656 (+ 32us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.999697 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":18,"thread_start_us":93,"threads_started":1}
I20260923 15:15:18.999917 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:18.995326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:40207 (local address 127.0.0.1:52668)
0923 15:15:18.995975 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:18.995983 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:18.996029 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:18.996341 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:18.996342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:18.996344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:18.996376 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.996377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.997723 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.997724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.998642 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:18.998645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:18.999644 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:18.999645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:18.999696 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:18.999699 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:18.999744 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:15:18.999776 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":8,"thread_start_us":85,"threads_started":1}
I20260923 15:15:19.001030 7811 tablet_service.cc:1467] Processing CreateTablet for tablet b026f89be4c34c41ab4e7cc063b13780 (DEFAULT_TABLE table=test-table [id=b68f106e20f44ad5ad65bc0bd4587144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:19.000977 7548 tablet_service.cc:1467] Processing CreateTablet for tablet b026f89be4c34c41ab4e7cc063b13780 (DEFAULT_TABLE table=test-table [id=b68f106e20f44ad5ad65bc0bd4587144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:19.001087 7679 tablet_service.cc:1467] Processing CreateTablet for tablet b026f89be4c34c41ab4e7cc063b13780 (DEFAULT_TABLE table=test-table [id=b68f106e20f44ad5ad65bc0bd4587144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:19.001381 7811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b026f89be4c34c41ab4e7cc063b13780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:19.001384 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b026f89be4c34c41ab4e7cc063b13780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:19.001384 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b026f89be4c34c41ab4e7cc063b13780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:19.004000 7892 tablet_bootstrap.cc:492] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Bootstrap starting.
I20260923 15:15:19.004729 7892 tablet_bootstrap.cc:654] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:19.004959 7894 tablet_bootstrap.cc:492] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: Bootstrap starting.
I20260923 15:15:19.004990 7893 tablet_bootstrap.cc:492] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: Bootstrap starting.
I20260923 15:15:19.005122 7892 log.cc:824] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:19.005611 7893 tablet_bootstrap.cc:654] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:19.005611 7894 tablet_bootstrap.cc:654] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:19.005977 7893 log.cc:824] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:19.005978 7894 log.cc:824] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:19.006551 7892 tablet_bootstrap.cc:492] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: No bootstrap required, opened a new log
I20260923 15:15:19.006629 7892 ts_tablet_manager.cc:1397] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 15:15:19.006803 7894 tablet_bootstrap.cc:492] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: No bootstrap required, opened a new log
I20260923 15:15:19.006878 7894 ts_tablet_manager.cc:1397] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 15:15:19.007108 7893 tablet_bootstrap.cc:492] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: No bootstrap required, opened a new log
I20260923 15:15:19.007172 7893 ts_tablet_manager.cc:1397] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 15:15:19.008844 7892 raft_consensus.cc:348] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.009006 7892 raft_consensus.cc:374] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:19.009052 7892 raft_consensus.cc:729] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcd79a2b89c48beb8c5677ab7e9b9e9, State: Initialized, Role: FOLLOWER
I20260923 15:15:19.009181 7892 consensus_queue.cc:260] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.009444 7892 ts_tablet_manager.cc:1428] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:15:19.009577 7878 heartbeater.cc:503] Master 127.21.204.126:34335 was elected leader, sending a full tablet report...
I20260923 15:15:19.009528 7894 raft_consensus.cc:348] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.009690 7894 raft_consensus.cc:374] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:19.009743 7894 raft_consensus.cc:729] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3bd97f0557d4953b75e190e64bbd2dc, State: Initialized, Role: FOLLOWER
I20260923 15:15:19.009697 7893 raft_consensus.cc:348] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.009859 7894 consensus_queue.cc:260] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.009944 7893 raft_consensus.cc:374] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:19.010004 7893 raft_consensus.cc:729] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bb4d41c6e84b3a9241ad2ad91cb73d, State: Initialized, Role: FOLLOWER
I20260923 15:15:19.010151 7894 ts_tablet_manager.cc:1428] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260923 15:15:19.010159 7893 consensus_queue.cc:260] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.010272 7614 heartbeater.cc:503] Master 127.21.204.126:34335 was elected leader, sending a full tablet report...
I20260923 15:15:19.010396 7893 ts_tablet_manager.cc:1428] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 15:15:19.010506 7745 heartbeater.cc:503] Master 127.21.204.126:34335 was elected leader, sending a full tablet report...
W20260923 15:15:19.080432 7746 tablet.cc:2369] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:15:19.174780 7901 raft_consensus.cc:482] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:15:19.174942 7901 raft_consensus.cc:504] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.175328 7901 leader_election.cc:288] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3bd97f0557d4953b75e190e64bbd2dc (127.21.204.65:42649), abcd79a2b89c48beb8c5677ab7e9b9e9 (127.21.204.67:45315)
I20260923 15:15:19.178419 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:19.175629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:40089 (local address 127.21.204.67:45315)
0923 15:15:19.175703 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:19.175706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:19.175714 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:15:19.175746 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:19.175746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:19.175756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:19.175831 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:19.175991 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.176483 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.177504 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.178067 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.178138 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:19.178146 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:19.178304 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 15:15:19.178346 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 15:15:19.178454 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:19.175522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:45315 (local address 127.21.204.66:40089)
0923 15:15:19.175589 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:19.175603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:19.175662 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:19.175836 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:19.175836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:19.175838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:19.175909 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:19.175911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.176566 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.176567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:19.177434 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:19.177437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.178151 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.178152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:19.178220 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:19.178224 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:19.178313 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 15:15:19.178362 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 15:15:19.178602 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:19.175649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:43743 (local address 127.21.204.65:42649)
0923 15:15:19.175845 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:19.175847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:19.175854 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:15:19.175878 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:19.175879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:19.175887 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:19.175956 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:19.176117 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.176620 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.177672 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.178239 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.178322 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:19.178330 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:19.178491 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 15:15:19.178541 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260923 15:15:19.178612 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:19.175540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:42649 (local address 127.21.204.66:43743)
0923 15:15:19.175702 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:19.175712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:19.175769 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:19.175962 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:19.175962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:19.175964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:19.176042 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:19.176044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.176710 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.176719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:19.177596 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:19.177599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.178345 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.178346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:19.178410 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:19.178413 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:19.178477 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 15:15:19.178517 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260923 15:15:19.178818 7831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b026f89be4c34c41ab4e7cc063b13780" candidate_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" is_pre_election: true
I20260923 15:15:19.179009 7831 raft_consensus.cc:2454] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bb4d41c6e84b3a9241ad2ad91cb73d in term 0.
I20260923 15:15:19.179067 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b026f89be4c34c41ab4e7cc063b13780" candidate_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" is_pre_election: true
I20260923 15:15:19.179239 7568 raft_consensus.cc:2454] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bb4d41c6e84b3a9241ad2ad91cb73d in term 0.
I20260923 15:15:19.179392 7634 leader_election.cc:302] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bb4d41c6e84b3a9241ad2ad91cb73d, abcd79a2b89c48beb8c5677ab7e9b9e9; no voters:
I20260923 15:15:19.179545 7901 raft_consensus.cc:2781] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:15:19.179632 7901 raft_consensus.cc:482] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:15:19.179668 7901 raft_consensus.cc:3037] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:19.180365 7901 raft_consensus.cc:504] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.180511 7901 leader_election.cc:288] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [CANDIDATE]: Term 1 election: Requested vote from peers b3bd97f0557d4953b75e190e64bbd2dc (127.21.204.65:42649), abcd79a2b89c48beb8c5677ab7e9b9e9 (127.21.204.67:45315)
I20260923 15:15:19.180666 7831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b026f89be4c34c41ab4e7cc063b13780" candidate_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9"
I20260923 15:15:19.180662 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b026f89be4c34c41ab4e7cc063b13780" candidate_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3bd97f0557d4953b75e190e64bbd2dc"
I20260923 15:15:19.180748 7568 raft_consensus.cc:3037] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:19.180773 7831 raft_consensus.cc:3037] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:19.181454 7568 raft_consensus.cc:2454] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bb4d41c6e84b3a9241ad2ad91cb73d in term 1.
I20260923 15:15:19.181557 7831 raft_consensus.cc:2454] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bb4d41c6e84b3a9241ad2ad91cb73d in term 1.
I20260923 15:15:19.181630 7632 leader_election.cc:302] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bb4d41c6e84b3a9241ad2ad91cb73d, b3bd97f0557d4953b75e190e64bbd2dc; no voters:
I20260923 15:15:19.181756 7901 raft_consensus.cc:2781] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:19.181959 7901 raft_consensus.cc:686] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 1 LEADER]: Becoming Leader. State: Replica: 12bb4d41c6e84b3a9241ad2ad91cb73d, State: Running, Role: LEADER
I20260923 15:15:19.182066 7901 consensus_queue.cc:237] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } }
I20260923 15:15:19.182862 7426 catalog_manager.cc:5495] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d reported cstate change: term changed from 0 to 1, leader changed from <none> to 12bb4d41c6e84b3a9241ad2ad91cb73d (127.21.204.66). New cstate: current_term: 1 leader_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bb4d41c6e84b3a9241ad2ad91cb73d" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 40207 } health_report { overall_health: HEALTHY } } }
W20260923 15:15:19.205344 7615 tablet.cc:2369] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:15:19.218624 7879 tablet.cc:2369] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:15:19.226944 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:19.224705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.21.204.66:40207)
0923 15:15:19.224786 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:19.224788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:19.224797 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:19.224842 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:19.224843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:19.224868 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:19.224933 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:19.225056 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.225543 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.226307 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:19.226489 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:19.226544 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:19.226667 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:19.226765 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:19.226826 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:19.226856 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 15:15:19.226872 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 15:15:19.228744 7568 raft_consensus.cc:1261] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 1 FOLLOWER]: Refusing update from remote peer 12bb4d41c6e84b3a9241ad2ad91cb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:19.228744 7831 raft_consensus.cc:1261] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 1 FOLLOWER]: Refusing update from remote peer 12bb4d41c6e84b3a9241ad2ad91cb73d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:19.229076 7901 consensus_queue.cc:1035] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "abcd79a2b89c48beb8c5677ab7e9b9e9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:19.229247 7903 consensus_queue.cc:1035] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3bd97f0557d4953b75e190e64bbd2dc" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:19.231822 7907 mvcc.cc:205] Tried to move back new op lower bound from 7332563022758010880 to 7332563022570303488. Current Snapshot: MvccSnapshot[applied={T|T < 7332563022758010880}]
I20260923 15:15:19.231827 7908 mvcc.cc:205] Tried to move back new op lower bound from 7332563022758010880 to 7332563022570303488. Current Snapshot: MvccSnapshot[applied={T|T < 7332563022758010880}]
I20260923 15:15:19.232332 7906 mvcc.cc:205] Tried to move back new op lower bound from 7332563022758010880 to 7332563022570303488. Current Snapshot: MvccSnapshot[applied={T|T < 7332563022758010880 or (T in {7332563022758010880})}]
I20260923 15:15:19.233906 7425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:19.234012 7425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:19.234793 7425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd8d381a8c4b427eaad621b26780e20d: Sending DeleteTablet for 3 replicas of tablet b026f89be4c34c41ab4e7cc063b13780
I20260923 15:15:19.235208 7811 tablet_service.cc:1514] Processing DeleteTablet for tablet b026f89be4c34c41ab4e7cc063b13780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:19 UTC) from {username='slave'} at 127.0.0.1:48516
I20260923 15:15:19.235447 7916 tablet_replica.cc:330] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: stopping tablet replica
I20260923 15:15:19.235435 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet b026f89be4c34c41ab4e7cc063b13780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:19 UTC) from {username='slave'} at 127.0.0.1:59378
I20260923 15:15:19.235541 7916 raft_consensus.cc:2229] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:19.235653 7916 raft_consensus.cc:2258] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:19.235726 7917 tablet_replica.cc:330] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc: stopping tablet replica
I20260923 15:15:19.235877 7917 raft_consensus.cc:2229] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:19.235872 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 7486
I20260923 15:15:19.236033 7917 raft_consensus.cc:2258] T b026f89be4c34c41ab4e7cc063b13780 P b3bd97f0557d4953b75e190e64bbd2dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:19.236078 7916 ts_tablet_manager.cc:1905] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:19.238040 7916 ts_tablet_manager.cc:1918] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:19.238119 7916 log.cc:1196] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/b026f89be4c34c41ab4e7cc063b13780
I20260923 15:15:19.238363 7916 ts_tablet_manager.cc:1939] T b026f89be4c34c41ab4e7cc063b13780 P abcd79a2b89c48beb8c5677ab7e9b9e9: Deleting consensus metadata
I20260923 15:15:19.238951 7412 catalog_manager.cc:4841] TS abcd79a2b89c48beb8c5677ab7e9b9e9 (127.21.204.67:45315): tablet b026f89be4c34c41ab4e7cc063b13780 (table test-table [id=b68f106e20f44ad5ad65bc0bd4587144]) successfully deleted
I20260923 15:15:19.239797 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet b026f89be4c34c41ab4e7cc063b13780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:19 UTC) from {username='slave'} at 127.0.0.1:52668
W20260923 15:15:19.243037 7410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:42649 (error 108)
I20260923 15:15:19.243160 7918 tablet_replica.cc:330] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d: stopping tablet replica
I20260923 15:15:19.243252 7918 raft_consensus.cc:2229] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:15:19.243261 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 7617
I20260923 15:15:19.243350 7918 raft_consensus.cc:2258] T b026f89be4c34c41ab4e7cc063b13780 P 12bb4d41c6e84b3a9241ad2ad91cb73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:19.244586 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:19.244139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:42649 (local address 127.0.0.1:59392)
0923 15:15:19.244503 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:19.244546 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
W20260923 15:15:19.245076 7410 catalog_manager.cc:4584] TS b3bd97f0557d4953b75e190e64bbd2dc (127.21.204.65:42649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b026f89be4c34c41ab4e7cc063b13780: Network error: Client connection negotiation failed: client connection to 127.21.204.65:42649: connect: Connection refused (error 111)
I20260923 15:15:19.248556 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 7748
I20260923 15:15:19.248610 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:19.248480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:40207 (local address 127.0.0.1:52678)
0923 15:15:19.248536 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:19.248567 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.66:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 15:15:19.253262 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 7395
2026-09-23T15:15:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (879 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:19 dist-test-slave-t5lt krb5kdc[7922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:19 dist-test-slave-t5lt krb5kdc[7922](info): set up 2 sockets
Sep 23 15:15:19 dist-test-slave-t5lt krb5kdc[7922](info): commencing operation
krb5kdc: starting...
W20260923 15:15:20.291411 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:20 dist-test-slave-t5lt krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:20.407052 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42767
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:42767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:20.487195 7935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:20.487397 7935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:20.487421 7935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:20.488565 7935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:20.488606 7935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:20.488620 7935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:20.488633 7935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:20.488683 7935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:20.490142 7935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:42767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:20.490391 7935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:20.490613 7935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:20.495568 7943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:20.495622 7935 server_base.cc:1034] running on GCE node
W20260923 15:15:20.495625 7941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:20.495568 7940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:20.496047 7935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:20.496347 7935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:20.497525 7935 hybrid_clock.cc:648] HybridClock initialized: now 1790176520497524 us; error 50 us; skew 500 ppm
Sep 23 15:15:20 dist-test-slave-t5lt krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:20.500036 7935 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:20.500524 7935 webserver.cc:460] Webserver started at http://127.21.204.126:33831/ using document root <none> and password file <none>
I20260923 15:15:20.500746 7935 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:20.500818 7935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:20.500970 7935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:20.502122 7935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "7725958136204a03ab606b57cd093bb7"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "c51e41cfa0aefe36315e4360ab6db663"
server_key_iv: "de3a30af3617522bab489062deb1a1ea"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.502481 7935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "7725958136204a03ab606b57cd093bb7"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "c51e41cfa0aefe36315e4360ab6db663"
server_key_iv: "de3a30af3617522bab489062deb1a1ea"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.503764 7935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:20.504552 7950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.504729 7935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:20.504833 7935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "7725958136204a03ab606b57cd093bb7"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "c51e41cfa0aefe36315e4360ab6db663"
server_key_iv: "de3a30af3617522bab489062deb1a1ea"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.504917 7935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:20.526955 7935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:20.529034 7935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:20.529201 7935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:20.533334 7935 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:42767
I20260923 15:15:20.533375 8002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:42767 every 8 connection(s)
I20260923 15:15:20.533748 7935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:20.534706 8003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:20.536943 8003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7: Bootstrap starting.
I20260923 15:15:20.537483 8003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:20.537711 8003 log.cc:824] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:20.538400 8003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7: No bootstrap required, opened a new log
I20260923 15:15:20.539942 8003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725958136204a03ab606b57cd093bb7" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42767 } }
I20260923 15:15:20.540073 8003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:20.540095 8003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7725958136204a03ab606b57cd093bb7, State: Initialized, Role: FOLLOWER
I20260923 15:15:20.540230 8003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725958136204a03ab606b57cd093bb7" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42767 } }
I20260923 15:15:20.540318 8003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:20.540340 8003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:20.540367 8003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:20.540839 8003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725958136204a03ab606b57cd093bb7" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42767 } }
I20260923 15:15:20.540921 8003 leader_election.cc:302] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7725958136204a03ab606b57cd093bb7; no voters:
I20260923 15:15:20.541041 8003 leader_election.cc:288] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:20.541113 8006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:20.541286 8006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [term 1 LEADER]: Becoming Leader. State: Replica: 7725958136204a03ab606b57cd093bb7, State: Running, Role: LEADER
I20260923 15:15:20.541308 8003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:20.541379 8006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725958136204a03ab606b57cd093bb7" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42767 } }
I20260923 15:15:20.541913 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 7935
I20260923 15:15:20.541932 8007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7725958136204a03ab606b57cd093bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725958136204a03ab606b57cd093bb7" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42767 } } }
I20260923 15:15:20.542007 8007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:20.542032 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:20.542245 22321 external_mini_cluster.cc:1428] Setting key ef346be58a84d41c1b74694a81479c49
I20260923 15:15:20.542428 8016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:20.542917 8008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7725958136204a03ab606b57cd093bb7. Latest consensus state: current_term: 1 leader_uuid: "7725958136204a03ab606b57cd093bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725958136204a03ab606b57cd093bb7" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42767 } } }
I20260923 15:15:20.543043 8008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:20.543139 8016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:20.544831 8016 catalog_manager.cc:1347] Generated new cluster ID: ca6f3b3ba34d4292a93dd42f380c2170
I20260923 15:15:20.544884 8016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:20 dist-test-slave-t5lt krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:20.552553 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:20.542824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.21.204.126:42767)
0923 15:15:20.543020 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:20.543029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:20.543063 (+ 34us) server_negotiation.cc:407] Connection header received
0923 15:15:20.543205 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:20.543212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:20.543275 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:20.544179 (+ 904us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:20.544269 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.545364 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.546244 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.546657 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.546794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:20.549256 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:20.549261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:20.549261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:20.549273 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:20.551004 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:20.551393 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:20.551396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:20.551396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:20.551497 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:20.551708 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:20.551709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:20.551709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:20.551950 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:20.552113 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:20.552209 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 15:15:20.552349 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260923 15:15:20.564889 8016 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:20.565426 8016 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:20.572962 8016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7: Generated new TSK 0
I20260923 15:15:20.573149 8016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:15:20.602897 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42767
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:20.699319 8030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:20.699524 8030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:20.699548 8030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:20.700634 8030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:20.700670 8030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:20.700695 8030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:15:20.702154 8030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:20.702327 8030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:20.702507 8030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:20.707576 8036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:20.707653 8030 server_base.cc:1034] running on GCE node
W20260923 15:15:20.707646 8035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:20.707582 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:20.707922 8030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:20.708163 8030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:20.709324 8030 hybrid_clock.cc:648] HybridClock initialized: now 1790176520709312 us; error 27 us; skew 500 ppm
Sep 23 15:15:20 dist-test-slave-t5lt krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:20.711949 8030 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:15:20.712390 8030 webserver.cc:460] Webserver started at http://127.21.204.65:42357/ using document root <none> and password file <none>
I20260923 15:15:20.712574 8030 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:20.712613 8030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:20.712702 8030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:20.713657 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "7bce09795dd641bcbb3972c1cd884c7e"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "cbe7547d9e6763ae4ef52cd537162b48"
server_key_iv: "f41cdd5f9b9123290000220dff569c67"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.714013 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "7bce09795dd641bcbb3972c1cd884c7e"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "cbe7547d9e6763ae4ef52cd537162b48"
server_key_iv: "f41cdd5f9b9123290000220dff569c67"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.715255 8030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:20.715942 8045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.716145 8030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:15:20.716240 8030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "7bce09795dd641bcbb3972c1cd884c7e"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "cbe7547d9e6763ae4ef52cd537162b48"
server_key_iv: "f41cdd5f9b9123290000220dff569c67"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.716326 8030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:20.736193 8030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:20.738503 8030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:20.738677 8030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:20.738936 8030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:20.739284 8030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:20.739323 8030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.739348 8030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:20.739362 8030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.745628 8030 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:37351
I20260923 15:15:20.745744 8158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:37351 every 8 connection(s)
I20260923 15:15:20.746055 8030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:15:20.748251 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 8030
I20260923 15:15:20.748368 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:15:20.748579 22321 external_mini_cluster.cc:1428] Setting key e1cd7e57b44d498464df06ff1d3c0162
Sep 23 15:15:20 dist-test-slave-t5lt krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:20.756542 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:20.746766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:33911 (local address 127.21.204.126:42767)
0923 15:15:20.746828 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:20.746829 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:20.747052 (+ 223us) server_negotiation.cc:407] Connection header received
0923 15:15:20.747638 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:20.747639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:20.747657 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:20.747718 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:20.748948 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.749598 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.750728 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.750944 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.751030 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:20.753422 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:20.753424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:20.753425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:20.753432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:20.755156 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:20.755550 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:20.755551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:20.755552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:20.755618 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:20.755858 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:20.755860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:20.755860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:20.756006 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:20.756085 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:20.756324 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 15:15:20.756414 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 15:15:20.756582 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:20.746748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42767 (local address 127.21.204.65:33911)
0923 15:15:20.746935 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:20.746970 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:20.747517 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:20.747788 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:20.747792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:20.748575 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:20.748847 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:20.748852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.749698 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.749699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:20.750608 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:20.750611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.751045 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.751046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:20.751177 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:20.751734 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:20.751745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:20.753321 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:20.755249 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:20.755256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:20.755258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:20.755468 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:20.755687 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:20.755688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:20.755689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:20.755789 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:20.756077 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:20.756081 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:20.756189 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 15:15:20.756270 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 15:15:20.756387 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260923 15:15:20.757229 8159 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42767
I20260923 15:15:20.757319 8159 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:20.757491 8159 heartbeater.cc:511] Master 127.21.204.126:42767 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:15:20.758095 7967 ts_manager.cc:194] Registered new tserver with Master: 7bce09795dd641bcbb3972c1cd884c7e (127.21.204.65:37351)
I20260923 15:15:20.758780 7967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:33911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:15:20.776412 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42767
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:20.857410 8164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:20.857600 8164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:20.857625 8164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:20.858810 8164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:20.858850 8164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:20.858877 8164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:15:20.860294 8164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:20.860540 8164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:20.860786 8164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:20.865643 8169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:20.865617 8170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:20.865615 8172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:20.865674 8164 server_base.cc:1034] running on GCE node
I20260923 15:15:20.866019 8164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:20.866204 8164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:20.867349 8164 hybrid_clock.cc:648] HybridClock initialized: now 1790176520867336 us; error 29 us; skew 500 ppm
Sep 23 15:15:20 dist-test-slave-t5lt krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:20.870155 8164 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:15:20.870582 8164 webserver.cc:460] Webserver started at http://127.21.204.66:40047/ using document root <none> and password file <none>
I20260923 15:15:20.870828 8164 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:20.870898 8164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:20.871048 8164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:20.872036 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "e9b429c0766848cda559174a3abfcf26"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "dacbd23c338666551c3e989f0d623438"
server_key_iv: "b083fb2bc236396404dc103ffc809d3d"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.872402 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "e9b429c0766848cda559174a3abfcf26"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "dacbd23c338666551c3e989f0d623438"
server_key_iv: "b083fb2bc236396404dc103ffc809d3d"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.873737 8164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:20.874541 8179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.874719 8164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.874823 8164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "e9b429c0766848cda559174a3abfcf26"
format_stamp: "Formatted at 2026-09-23 15:15:20 on dist-test-slave-t5lt"
server_key: "dacbd23c338666551c3e989f0d623438"
server_key_iv: "b083fb2bc236396404dc103ffc809d3d"
server_key_version: "encryptionkey@0"
I20260923 15:15:20.874908 8164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:20.890081 8164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:20.892089 8164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:20.892359 8164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:20.892612 8164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:20.892925 8164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:20.892961 8164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.892982 8164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:20.893028 8164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:20.898727 8164 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:36579
I20260923 15:15:20.898777 8292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:36579 every 8 connection(s)
I20260923 15:15:20.899070 8164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:15:20.901176 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 8164
I20260923 15:15:20.901324 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:15:20.901528 22321 external_mini_cluster.cc:1428] Setting key f0e1f81619ac4c7f3614b2b527481e12
Sep 23 15:15:20 dist-test-slave-t5lt krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:20.909782 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:20.899765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:53849 (local address 127.21.204.126:42767)
0923 15:15:20.899829 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:20.899831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:20.900035 (+ 204us) server_negotiation.cc:407] Connection header received
0923 15:15:20.900531 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:20.900532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:20.900544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:20.900611 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:20.901945 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.902538 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.904037 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.904350 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.904425 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:20.906776 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:20.906778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:20.906778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:20.906787 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:20.908395 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:20.908815 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:20.908816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:20.908817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:20.908880 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:20.909135 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:20.909136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:20.909136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:20.909252 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:20.909325 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:20.909613 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 15:15:20.909694 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 15:15:20.910190 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:20.899740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42767 (local address 127.21.204.66:53849)
0923 15:15:20.899907 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:20.899947 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:20.900415 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:20.900686 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:20.900689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:20.901547 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:20.901812 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:20.901817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.902627 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.902629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:20.903944 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:20.903947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:20.904441 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:20.904442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:20.904551 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:20.905061 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:20.905070 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:20.906674 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:20.908497 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:20.908504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:20.908506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:20.908727 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:20.908945 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:20.908946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:20.908947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:20.909064 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:20.909330 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:20.909338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:20.909480 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 15:15:20.909580 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 15:15:20.909736 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
I20260923 15:15:20.911100 8293 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42767
I20260923 15:15:20.911182 8293 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:20.911484 8293 heartbeater.cc:511] Master 127.21.204.126:42767 requested a full tablet report, sending...
I20260923 15:15:20.911907 7967 ts_manager.cc:194] Registered new tserver with Master: e9b429c0766848cda559174a3abfcf26 (127.21.204.66:36579)
I20260923 15:15:20.912235 7967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:53849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:15:20.929574 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42767
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:21.015646 8298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:21.015872 8298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:21.015898 8298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:21.017177 8298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:21.017223 8298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:21.017253 8298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:15:21.019018 8298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:21.019277 8298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:21.019563 8298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:21.025552 8303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.025564 8306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.025561 8304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:21.025700 8298 server_base.cc:1034] running on GCE node
I20260923 15:15:21.025959 8298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:21.026211 8298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:21.027391 8298 hybrid_clock.cc:648] HybridClock initialized: now 1790176521027356 us; error 46 us; skew 500 ppm
Sep 23 15:15:21 dist-test-slave-t5lt krb5kdc[7922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:21.033591 8298 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:15:21.034099 8298 webserver.cc:460] Webserver started at http://127.21.204.67:41779/ using document root <none> and password file <none>
I20260923 15:15:21.034344 8298 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:21.034389 8298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:21.034526 8298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:21.035545 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "e27431e7adf84a1989d1bddbbe7bdcc3"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "fbe72bda56bb191b1fb17138963a9fe0"
server_key_iv: "13fc2eaa48c791eacab12227402daf87"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.035894 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "e27431e7adf84a1989d1bddbbe7bdcc3"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "fbe72bda56bb191b1fb17138963a9fe0"
server_key_iv: "13fc2eaa48c791eacab12227402daf87"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.037181 8298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:21.037974 8313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.038152 8298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:21.038254 8298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "e27431e7adf84a1989d1bddbbe7bdcc3"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "fbe72bda56bb191b1fb17138963a9fe0"
server_key_iv: "13fc2eaa48c791eacab12227402daf87"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.038327 8298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:21.059185 8298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:21.061647 8298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:21.061822 8298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:21.062136 8298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:21.062464 8298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:21.062520 8298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.062563 8298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:21.062595 8298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.068454 8298 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:43857
I20260923 15:15:21.068661 8426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:43857 every 8 connection(s)
I20260923 15:15:21.068871 8298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:15:21.074662 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 8298
I20260923 15:15:21.074805 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:15:21.074987 22321 external_mini_cluster.cc:1428] Setting key d1cd01f07c913331359b5b12bc10b5ca
Sep 23 15:15:21 dist-test-slave-t5lt krb5kdc[7922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:21.078785 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.069637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:58629 (local address 127.21.204.126:42767)
0923 15:15:21.069705 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.069707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.069917 (+ 210us) server_negotiation.cc:407] Connection header received
0923 15:15:21.070448 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.070448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.070461 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.070519 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:21.071550 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.072067 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.073231 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.073498 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.073576 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.075945 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:21.075946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:21.075947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:21.075956 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:21.077218 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:21.077647 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:21.077648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:21.077648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:21.077699 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:21.078118 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:21.078119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:21.078119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:21.078234 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:21.078289 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.078645 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.078710 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:15:21.079015 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.069630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42767 (local address 127.21.204.67:58629)
0923 15:15:21.069776 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.069811 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.070359 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.070572 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.070575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.071201 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:21.071405 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.071410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.072164 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.072165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.073145 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.073148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.073585 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.073586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.073688 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.074205 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:21.074214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:21.075827 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:21.077289 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:21.077296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:21.077298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:21.077582 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:21.077794 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:21.077795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:21.077796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:21.078043 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:21.078311 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:21.078318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:21.078446 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.078617 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.078770 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260923 15:15:21.079615 8427 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42767
I20260923 15:15:21.079761 8427 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:21.079967 8427 heartbeater.cc:511] Master 127.21.204.126:42767 requested a full tablet report, sending...
I20260923 15:15:21.080412 7967 ts_manager.cc:194] Registered new tserver with Master: e27431e7adf84a1989d1bddbbe7bdcc3 (127.21.204.67:43857)
I20260923 15:15:21.080789 7967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:58629
I20260923 15:15:21.086709 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:15:21.094084 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.088021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.21.204.126:42767)
0923 15:15:21.088100 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.088102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.088164 (+ 62us) server_negotiation.cc:407] Connection header received
0923 15:15:21.088221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.088222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.088232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.088279 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:21.088801 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.089265 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.089984 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.090158 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.090208 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.091361 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:21.091362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:21.091362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:21.091370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:21.092740 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:21.093208 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:21.093209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:21.093210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:21.093267 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:21.093522 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:21.093522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:21.093522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:21.093651 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:21.093732 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.093850 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.093972 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 15:15:21.095795 7967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:15:21.097536 7967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:15:21.104281 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.100199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37351 (local address 127.0.0.1:52246)
0923 15:15:21.100519 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.100528 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.100581 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.100823 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.100824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.100840 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.100915 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.100917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.102280 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.102281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.103217 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.103219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.103953 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.103954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.104032 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.104046 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.104149 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.104205 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":5,"thread_start_us":105,"threads_started":1}
I20260923 15:15:21.104295 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.100324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.21.204.65:37351)
0923 15:15:21.100535 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.100540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.100573 (+ 33us) server_negotiation.cc:407] Connection header received
0923 15:15:21.100644 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.100646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.100659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.100743 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.101002 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.102149 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.103317 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.103857 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.103940 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.104002 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.104171 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.104214 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
I20260923 15:15:21.104542 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.100005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:36579 (local address 127.0.0.1:57176)
0923 15:15:21.100359 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.100370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.100422 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.100823 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.100824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.100827 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.100915 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.100918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.102257 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.102258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.103361 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.103363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.104321 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.104321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.104380 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.104384 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.104446 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.104485 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":38,"thread_start_us":78,"threads_started":1}
I20260923 15:15:21.104773 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.100215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.21.204.66:36579)
0923 15:15:21.100588 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.100592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.100600 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:15:21.100641 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.100643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.100659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.100735 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.101002 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.102149 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.103443 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.104163 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.104229 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.104287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.104630 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.104680 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":90,"threads_started":1}
I20260923 15:15:21.105072 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.100005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:43857 (local address 127.0.0.1:48810)
0923 15:15:21.100284 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.100335 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.100410 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.100818 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.100821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.100824 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.100913 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.100915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.101871 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.101893 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.103076 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.103079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.104126 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.104128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.104201 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.104207 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.104260 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.104307 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":4,"thread_start_us":50,"threads_started":1}
I20260923 15:15:21.105036 8093 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0c639edc4643e8b4f7429d311d13b5 (DEFAULT_TABLE table=test-table [id=8268f193a9924cea80d1a8d27660e273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:21.105254 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.100094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.21.204.67:43857)
0923 15:15:21.100564 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.100569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.100579 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:15:21.100617 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.100619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.100636 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.100707 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.101021 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.101771 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.103165 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.103899 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.103970 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.104805 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.104850 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.104904 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":93,"threads_started":1}
I20260923 15:15:21.105449 8093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0c639edc4643e8b4f7429d311d13b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:21.105419 8227 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0c639edc4643e8b4f7429d311d13b5 (DEFAULT_TABLE table=test-table [id=8268f193a9924cea80d1a8d27660e273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:21.105702 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0c639edc4643e8b4f7429d311d13b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:21.105895 8361 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0c639edc4643e8b4f7429d311d13b5 (DEFAULT_TABLE table=test-table [id=8268f193a9924cea80d1a8d27660e273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:21.106190 8361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0c639edc4643e8b4f7429d311d13b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:21.108408 8444 tablet_bootstrap.cc:492] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Bootstrap starting.
I20260923 15:15:21.108866 8445 tablet_bootstrap.cc:492] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e: Bootstrap starting.
I20260923 15:15:21.109354 8444 tablet_bootstrap.cc:654] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:21.109354 8446 tablet_bootstrap.cc:492] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Bootstrap starting.
I20260923 15:15:21.109436 8445 tablet_bootstrap.cc:654] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:21.110026 8446 tablet_bootstrap.cc:654] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:21.110049 8444 log.cc:824] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:21.110052 8445 log.cc:824] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:21.110616 8446 log.cc:824] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:21.110980 8444 tablet_bootstrap.cc:492] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: No bootstrap required, opened a new log
I20260923 15:15:21.111071 8444 ts_tablet_manager.cc:1397] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 15:15:21.111143 8445 tablet_bootstrap.cc:492] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e: No bootstrap required, opened a new log
I20260923 15:15:21.111217 8445 ts_tablet_manager.cc:1397] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:21.111476 8446 tablet_bootstrap.cc:492] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: No bootstrap required, opened a new log
I20260923 15:15:21.111542 8446 ts_tablet_manager.cc:1397] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 15:15:21.113189 8445 raft_consensus.cc:348] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.113374 8445 raft_consensus.cc:374] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:21.113413 8445 raft_consensus.cc:729] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bce09795dd641bcbb3972c1cd884c7e, State: Initialized, Role: FOLLOWER
I20260923 15:15:21.113536 8445 consensus_queue.cc:260] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.113562 8446 raft_consensus.cc:348] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.113497 8444 raft_consensus.cc:348] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.113722 8446 raft_consensus.cc:374] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:21.113724 8444 raft_consensus.cc:374] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:21.113760 8446 raft_consensus.cc:729] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27431e7adf84a1989d1bddbbe7bdcc3, State: Initialized, Role: FOLLOWER
I20260923 15:15:21.113782 8444 raft_consensus.cc:729] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b429c0766848cda559174a3abfcf26, State: Initialized, Role: FOLLOWER
I20260923 15:15:21.113837 8445 ts_tablet_manager.cc:1428] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:15:21.113869 8446 consensus_queue.cc:260] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.113987 8159 heartbeater.cc:503] Master 127.21.204.126:42767 was elected leader, sending a full tablet report...
I20260923 15:15:21.113977 8444 consensus_queue.cc:260] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.114146 8446 ts_tablet_manager.cc:1428] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 15:15:21.114275 8427 heartbeater.cc:503] Master 127.21.204.126:42767 was elected leader, sending a full tablet report...
I20260923 15:15:21.114681 8444 ts_tablet_manager.cc:1428] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 15:15:21.114856 8293 heartbeater.cc:503] Master 127.21.204.126:42767 was elected leader, sending a full tablet report...
W20260923 15:15:21.149638 8294 tablet.cc:2369] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 15:15:21.246805 8160 tablet.cc:2369] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:15:21.256596 8452 raft_consensus.cc:482] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:15:21.256765 8452 raft_consensus.cc:504] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.257210 8452 leader_election.cc:288] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e27431e7adf84a1989d1bddbbe7bdcc3 (127.21.204.67:43857), 7bce09795dd641bcbb3972c1cd884c7e (127.21.204.65:37351)
I20260923 15:15:21.261127 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.257636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:39591 (local address 127.21.204.65:37351)
0923 15:15:21.257695 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.257697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.257746 (+ 49us) server_negotiation.cc:407] Connection header received
0923 15:15:21.257819 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.257820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.257830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.257948 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.258083 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.258753 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.260211 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.260753 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.260833 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.260844 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.261017 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.261058 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 15:15:21.261127 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.257514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:38481 (local address 127.21.204.67:43857)
0923 15:15:21.257595 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.257597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.257607 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:15:21.257639 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.257640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.257651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.257718 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.257910 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.258693 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.260212 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.260790 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.260854 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.260861 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.261018 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.261069 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 15:15:21.261168 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.257395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:43857 (local address 127.21.204.66:38481)
0923 15:15:21.257481 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.257495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.257579 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.257727 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.257728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.257729 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.257807 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.257809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.258773 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.258773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.260110 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.260113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.260863 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.260864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.260937 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.260938 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.260997 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.261053 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":70}
I20260923 15:15:21.261173 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.257532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37351 (local address 127.21.204.66:39591)
0923 15:15:21.257687 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.257697 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.257752 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.257963 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.257964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.257965 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.258019 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.258020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.258824 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.258825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.260110 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.260113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.260856 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.260858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.260933 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.260938 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.260997 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.261053 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":61,"thread_start_us":96,"threads_started":1}
I20260923 15:15:21.261597 8381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c639edc4643e8b4f7429d311d13b5" candidate_uuid: "e9b429c0766848cda559174a3abfcf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" is_pre_election: true
I20260923 15:15:21.261600 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c639edc4643e8b4f7429d311d13b5" candidate_uuid: "e9b429c0766848cda559174a3abfcf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bce09795dd641bcbb3972c1cd884c7e" is_pre_election: true
I20260923 15:15:21.261756 8381 raft_consensus.cc:2454] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b429c0766848cda559174a3abfcf26 in term 0.
I20260923 15:15:21.261756 8113 raft_consensus.cc:2454] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b429c0766848cda559174a3abfcf26 in term 0.
I20260923 15:15:21.261984 8181 leader_election.cc:302] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bce09795dd641bcbb3972c1cd884c7e, e9b429c0766848cda559174a3abfcf26; no voters:
I20260923 15:15:21.262105 8452 raft_consensus.cc:2781] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:15:21.262198 8452 raft_consensus.cc:482] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:15:21.262235 8452 raft_consensus.cc:3037] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:21.262944 8452 raft_consensus.cc:504] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.263082 8452 leader_election.cc:288] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [CANDIDATE]: Term 1 election: Requested vote from peers e27431e7adf84a1989d1bddbbe7bdcc3 (127.21.204.67:43857), 7bce09795dd641bcbb3972c1cd884c7e (127.21.204.65:37351)
I20260923 15:15:21.263463 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c639edc4643e8b4f7429d311d13b5" candidate_uuid: "e9b429c0766848cda559174a3abfcf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bce09795dd641bcbb3972c1cd884c7e"
I20260923 15:15:21.263463 8381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0c639edc4643e8b4f7429d311d13b5" candidate_uuid: "e9b429c0766848cda559174a3abfcf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3"
I20260923 15:15:21.263552 8381 raft_consensus.cc:3037] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:21.263552 8113 raft_consensus.cc:3037] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:21.264443 8381 raft_consensus.cc:2454] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b429c0766848cda559174a3abfcf26 in term 1.
I20260923 15:15:21.264443 8113 raft_consensus.cc:2454] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b429c0766848cda559174a3abfcf26 in term 1.
I20260923 15:15:21.264634 8181 leader_election.cc:302] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bce09795dd641bcbb3972c1cd884c7e, e9b429c0766848cda559174a3abfcf26; no voters:
I20260923 15:15:21.264724 8452 raft_consensus.cc:2781] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:21.264887 8452 raft_consensus.cc:686] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 1 LEADER]: Becoming Leader. State: Replica: e9b429c0766848cda559174a3abfcf26, State: Running, Role: LEADER
I20260923 15:15:21.264997 8452 consensus_queue.cc:237] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } }
I20260923 15:15:21.265837 7967 catalog_manager.cc:5495] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9b429c0766848cda559174a3abfcf26 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "e9b429c0766848cda559174a3abfcf26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9b429c0766848cda559174a3abfcf26" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 36579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 } health_report { overall_health: UNKNOWN } } }
I20260923 15:15:21.285593 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.283435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.21.204.66:36579)
0923 15:15:21.283496 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.283498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.283506 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:15:21.283562 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.283563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.283578 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.283630 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:21.283741 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.284206 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.284917 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.285094 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.285142 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.285255 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:21.285337 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:21.285403 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.285453 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.285463 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 15:15:21.287678 8381 raft_consensus.cc:1261] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 1 FOLLOWER]: Refusing update from remote peer e9b429c0766848cda559174a3abfcf26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:21.287742 8113 raft_consensus.cc:1261] T 4e0c639edc4643e8b4f7429d311d13b5 P 7bce09795dd641bcbb3972c1cd884c7e [term 1 FOLLOWER]: Refusing update from remote peer e9b429c0766848cda559174a3abfcf26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:21.288089 8454 consensus_queue.cc:1035] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27431e7adf84a1989d1bddbbe7bdcc3" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:21.288252 8452 consensus_queue.cc:1035] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bce09795dd641bcbb3972c1cd884c7e" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:21.291239 8458 mvcc.cc:205] Tried to move back new op lower bound from 7332563031190773760 to 7332563031102062592. Current Snapshot: MvccSnapshot[applied={T|T < 7332563031190773760}]
I20260923 15:15:21.293536 7967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:21.293641 7967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:21.294582 7967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7725958136204a03ab606b57cd093bb7: Sending DeleteTablet for 3 replicas of tablet 4e0c639edc4643e8b4f7429d311d13b5
I20260923 15:15:21.294940 8227 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0c639edc4643e8b4f7429d311d13b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:21 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:57176
I20260923 15:15:21.295168 8467 tablet_replica.cc:330] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: stopping tablet replica
I20260923 15:15:21.295282 8467 raft_consensus.cc:2229] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:15:21.295400 8361 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0c639edc4643e8b4f7429d311d13b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:21 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:48810
I20260923 15:15:21.295435 8467 raft_consensus.cc:2258] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:21.295418 8093 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e0c639edc4643e8b4f7429d311d13b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:21 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:52246
I20260923 15:15:21.295774 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 8030
I20260923 15:15:21.295895 8467 ts_tablet_manager.cc:1905] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:21.297991 8467 ts_tablet_manager.cc:1918] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:21.298067 8467 log.cc:1196] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/4e0c639edc4643e8b4f7429d311d13b5
I20260923 15:15:21.298331 8467 ts_tablet_manager.cc:1939] T 4e0c639edc4643e8b4f7429d311d13b5 P e9b429c0766848cda559174a3abfcf26: Deleting consensus metadata
I20260923 15:15:21.298770 7951 catalog_manager.cc:4841] TS e9b429c0766848cda559174a3abfcf26 (127.21.204.66:36579): tablet 4e0c639edc4643e8b4f7429d311d13b5 (table test-table [id=8268f193a9924cea80d1a8d27660e273]) successfully deleted
W20260923 15:15:21.302011 7952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:37351 (error 108)
I20260923 15:15:21.302186 8468 tablet_replica.cc:330] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: stopping tablet replica
I20260923 15:15:21.302323 8468 raft_consensus.cc:2229] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:21.302448 8468 raft_consensus.cc:2258] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:21.302603 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.302472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:37351 (local address 127.0.0.1:52260)
0923 15:15:21.302531 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.302570 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 15:15:21.302726 7952 catalog_manager.cc:4584] TS 7bce09795dd641bcbb3972c1cd884c7e (127.21.204.65:37351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e0c639edc4643e8b4f7429d311d13b5: Network error: Client connection negotiation failed: client connection to 127.21.204.65:37351: connect: Connection refused (error 111)
I20260923 15:15:21.302757 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 8164
I20260923 15:15:21.306166 8468 ts_tablet_manager.cc:1905] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:21.308043 8468 ts_tablet_manager.cc:1918] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:21.308120 8468 log.cc:1196] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/4e0c639edc4643e8b4f7429d311d13b5
I20260923 15:15:21.308391 8468 ts_tablet_manager.cc:1939] T 4e0c639edc4643e8b4f7429d311d13b5 P e27431e7adf84a1989d1bddbbe7bdcc3: Deleting consensus metadata
I20260923 15:15:21.308825 7952 catalog_manager.cc:4841] TS e27431e7adf84a1989d1bddbbe7bdcc3 (127.21.204.67:43857): tablet 4e0c639edc4643e8b4f7429d311d13b5 (table test-table [id=8268f193a9924cea80d1a8d27660e273]) successfully deleted
I20260923 15:15:21.309171 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 8298
I20260923 15:15:21.314544 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 7935
2026-09-23T15:15:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2061 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T15:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:21Z Disabled control of system clock
I20260923 15:15:21.341717 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42769
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:38609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:42769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:21.424569 8474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:21.424768 8474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:21.424788 8474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:21.425980 8474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:21.426019 8474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:21.426033 8474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:21.426045 8474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:21.426057 8474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:21.427320 8474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:42769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:21.427501 8474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:21.427721 8474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:21.432799 8482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.432811 8480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.432816 8479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:21.433182 8474 server_base.cc:1034] running on GCE node
I20260923 15:15:21.433439 8474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:21.433728 8474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:21.434896 8474 hybrid_clock.cc:648] HybridClock initialized: now 1790176521434884 us; error 27 us; skew 500 ppm
I20260923 15:15:21.436172 8474 webserver.cc:460] Webserver started at http://127.21.204.126:37191/ using document root <none> and password file <none>
I20260923 15:15:21.436443 8474 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:21.436513 8474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:21.436662 8474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:21.437700 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "8307bb4501064dd5b8fb931ba265b2ad"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "dc0643021ee918cd6d9da20599fa1c81"
server_key_iv: "eb81c086bb7818af1dff40363ac5622a"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.438081 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "8307bb4501064dd5b8fb931ba265b2ad"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "dc0643021ee918cd6d9da20599fa1c81"
server_key_iv: "eb81c086bb7818af1dff40363ac5622a"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.439371 8474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:21.440099 8488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.440303 8474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:21.440403 8474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "8307bb4501064dd5b8fb931ba265b2ad"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "dc0643021ee918cd6d9da20599fa1c81"
server_key_iv: "eb81c086bb7818af1dff40363ac5622a"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.440488 8474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:21.487732 8474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:21.488102 8474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:21.488268 8474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:21.492467 8474 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:42769
I20260923 15:15:21.492511 8540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:42769 every 8 connection(s)
I20260923 15:15:21.492895 8474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:21.493937 8541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:21.496248 8541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad: Bootstrap starting.
I20260923 15:15:21.496349 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 8474
I20260923 15:15:21.496487 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:21.496676 22321 external_mini_cluster.cc:1428] Setting key f62c692834c332e747b7882fb3d036ab
I20260923 15:15:21.496989 8541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:21.497354 8541 log.cc:824] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:21.498128 8541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad: No bootstrap required, opened a new log
I20260923 15:15:21.500464 8541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8307bb4501064dd5b8fb931ba265b2ad" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42769 } }
I20260923 15:15:21.500624 8541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:21.500662 8541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8307bb4501064dd5b8fb931ba265b2ad, State: Initialized, Role: FOLLOWER
I20260923 15:15:21.500788 8541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8307bb4501064dd5b8fb931ba265b2ad" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42769 } }
I20260923 15:15:21.500861 8541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:21.500890 8541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:21.500937 8541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:21.501632 8541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8307bb4501064dd5b8fb931ba265b2ad" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42769 } }
I20260923 15:15:21.501742 8541 leader_election.cc:302] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8307bb4501064dd5b8fb931ba265b2ad; no voters:
I20260923 15:15:21.501942 8541 leader_election.cc:288] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:21.501982 8546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:21.502110 8546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [term 1 LEADER]: Becoming Leader. State: Replica: 8307bb4501064dd5b8fb931ba265b2ad, State: Running, Role: LEADER
I20260923 15:15:21.502202 8546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8307bb4501064dd5b8fb931ba265b2ad" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42769 } }
I20260923 15:15:21.502257 8541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:21.502518 8546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8307bb4501064dd5b8fb931ba265b2ad. Latest consensus state: current_term: 1 leader_uuid: "8307bb4501064dd5b8fb931ba265b2ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8307bb4501064dd5b8fb931ba265b2ad" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42769 } } }
I20260923 15:15:21.502597 8546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:21.502527 8547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8307bb4501064dd5b8fb931ba265b2ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8307bb4501064dd5b8fb931ba265b2ad" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42769 } } }
I20260923 15:15:21.502893 8547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:21.502944 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.497201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.21.204.126:42769)
0923 15:15:21.497408 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.497417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.497449 (+ 32us) server_negotiation.cc:407] Connection header received
0923 15:15:21.498023 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.498032 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.498107 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.498261 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:21.498480 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.499880 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.500635 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.501240 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.501382 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.502066 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:21.502070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:21.502071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:21.502278 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:21.502319 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:21.502334 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:21.502394 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.502597 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.502685 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260923 15:15:21.503290 8551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:21.503768 8551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:21.505260 8551 catalog_manager.cc:1347] Generated new cluster ID: f0ae8bd6b66d4bb8bdfc3b7c09675875
I20260923 15:15:21.505362 8551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:21.519675 8551 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:21.520341 8551 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:21.523816 8551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad: Generated new TSK 0
I20260923 15:15:21.523973 8551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:21.525828 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42769
--builtin_ntp_servers=127.21.204.84:38609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:21.628978 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:21.629236 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:21.629273 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:21.630507 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:21.630571 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:21.630614 8565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:15:21.631983 8565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:21.632242 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:21.632543 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:21.637504 8570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.637547 8571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:21.637655 8565 server_base.cc:1034] running on GCE node
W20260923 15:15:21.637521 8573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:21.638029 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:21.638294 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:21.639456 8565 hybrid_clock.cc:648] HybridClock initialized: now 1790176521639445 us; error 26 us; skew 500 ppm
I20260923 15:15:21.640694 8565 webserver.cc:460] Webserver started at http://127.21.204.65:40485/ using document root <none> and password file <none>
I20260923 15:15:21.640959 8565 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:21.641033 8565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:21.641188 8565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:21.642340 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "f18c041befb243a19d689dfe3e576edd"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "8a01e3e32464c4ca3524b4df391ae81c"
server_key_iv: "f63ccfccdad53fedecf4dfe24615dda1"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.642715 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "f18c041befb243a19d689dfe3e576edd"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "8a01e3e32464c4ca3524b4df391ae81c"
server_key_iv: "f63ccfccdad53fedecf4dfe24615dda1"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.644222 8565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 15:15:21.645018 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.645262 8565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:21.645359 8565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "f18c041befb243a19d689dfe3e576edd"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "8a01e3e32464c4ca3524b4df391ae81c"
server_key_iv: "f63ccfccdad53fedecf4dfe24615dda1"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.645429 8565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:21.657766 8565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:21.658072 8565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:21.658196 8565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:21.658394 8565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:21.658701 8565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:21.658738 8565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.658764 8565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:21.658788 8565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.665465 8565 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:41203
I20260923 15:15:21.665560 8692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:41203 every 8 connection(s)
I20260923 15:15:21.665896 8565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:15:21.670614 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.666631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:58621 (local address 127.21.204.126:42769)
0923 15:15:21.666677 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.666679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.666905 (+ 226us) server_negotiation.cc:407] Connection header received
0923 15:15:21.667327 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.667328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.667338 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.667386 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:21.667990 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.668470 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.669376 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.669555 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.669603 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.670392 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:21.670393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:21.670393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:21.670398 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:21.670411 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:21.670415 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:21.670458 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.670538 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.670558 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:15:21.670789 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.666614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42769 (local address 127.21.204.65:58621)
0923 15:15:21.666786 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.666817 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.667250 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.667454 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.667457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.667465 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:21.667927 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.667931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.668552 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.668552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.669305 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.669306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.669627 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.669628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.669727 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.670268 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:21.670268 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:21.670294 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:21.670315 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:21.670469 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:21.670474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:21.670476 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.670526 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.670599 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260923 15:15:21.671223 8693 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42769
I20260923 15:15:21.671325 8693 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:21.671361 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 8565
I20260923 15:15:21.671469 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:15:21.671540 8693 heartbeater.cc:511] Master 127.21.204.126:42769 requested a full tablet report, sending...
I20260923 15:15:21.671682 22321 external_mini_cluster.cc:1428] Setting key a02bc9c90e4eeee01f0e9ef51330c236
I20260923 15:15:21.672109 8505 ts_manager.cc:194] Registered new tserver with Master: f18c041befb243a19d689dfe3e576edd (127.21.204.65:41203)
I20260923 15:15:21.672732 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42769
--builtin_ntp_servers=127.21.204.84:38609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 15:15:21.672987 8505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.65:58621
W20260923 15:15:21.753655 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:21.753859 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:21.753929 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:21.755026 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:21.755060 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:21.755085 8697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:15:21.756325 8697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:21.756510 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:21.756736 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:21.761693 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.761700 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.761709 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:21.761830 8697 server_base.cc:1034] running on GCE node
I20260923 15:15:21.762156 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:21.762382 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:21.763527 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790176521763517 us; error 31 us; skew 500 ppm
I20260923 15:15:21.764830 8697 webserver.cc:460] Webserver started at http://127.21.204.66:37123/ using document root <none> and password file <none>
I20260923 15:15:21.765090 8697 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:21.765138 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:21.765329 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:21.766467 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "123d2085bc934f8d989807df0c7105ac"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "bd6309fa510747fce077d084faa31fd5"
server_key_iv: "c8adad7717caf4d874d29ecbfdb3f73e"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.766819 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "123d2085bc934f8d989807df0c7105ac"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "bd6309fa510747fce077d084faa31fd5"
server_key_iv: "c8adad7717caf4d874d29ecbfdb3f73e"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.768216 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 15:15:21.769043 8711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.769258 8697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:21.769399 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "123d2085bc934f8d989807df0c7105ac"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "bd6309fa510747fce077d084faa31fd5"
server_key_iv: "c8adad7717caf4d874d29ecbfdb3f73e"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.769479 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:21.782262 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:21.782547 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:21.782689 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:21.782908 8697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:21.783222 8697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:21.783277 8697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.783329 8697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:21.783356 8697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.789361 8697 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:39471
I20260923 15:15:21.789402 8825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:39471 every 8 connection(s)
I20260923 15:15:21.789748 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:15:21.794773 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.790590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:53671 (local address 127.21.204.126:42769)
0923 15:15:21.790665 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.790667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.790918 (+ 251us) server_negotiation.cc:407] Connection header received
0923 15:15:21.791383 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.791384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.791394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.791444 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:21.792060 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.792545 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.793463 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.793642 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.793690 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.794576 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:21.794577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:21.794577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:21.794581 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:21.794593 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:21.794597 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:21.794638 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.794698 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.794717 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:15:21.794960 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.790580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42769 (local address 127.21.204.66:53671)
0923 15:15:21.790794 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.790827 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.791299 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.791500 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.791503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.791512 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:21.791991 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.791994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.792627 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.792628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.793390 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.793391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.793716 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.793717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.793823 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.794456 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:21.794457 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:21.794484 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:21.794506 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:21.794638 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:21.794640 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:21.794643 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.794686 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.794760 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260923 15:15:21.795418 8826 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42769
I20260923 15:15:21.795496 8826 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:21.795635 8826 heartbeater.cc:511] Master 127.21.204.126:42769 requested a full tablet report, sending...
I20260923 15:15:21.795959 8505 ts_manager.cc:194] Registered new tserver with Master: 123d2085bc934f8d989807df0c7105ac (127.21.204.66:39471)
I20260923 15:15:21.796343 8505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.66:53671
I20260923 15:15:21.797623 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 8697
I20260923 15:15:21.797712 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:15:21.797955 22321 external_mini_cluster.cc:1428] Setting key 974923d07b2d6dd6ca5dfaaed08935ff
I20260923 15:15:21.798877 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42769
--builtin_ntp_servers=127.21.204.84:38609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:21.883450 8830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:21.883672 8830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:21.883708 8830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:21.884819 8830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:21.884882 8830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:21.884922 8830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:15:21.886255 8830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:21.886510 8830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:21.886790 8830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:21.891744 8836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.891780 8835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:21.891752 8838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:21.892190 8830 server_base.cc:1034] running on GCE node
I20260923 15:15:21.892386 8830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:21.892572 8830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:21.893714 8830 hybrid_clock.cc:648] HybridClock initialized: now 1790176521893702 us; error 27 us; skew 500 ppm
I20260923 15:15:21.894949 8830 webserver.cc:460] Webserver started at http://127.21.204.67:39449/ using document root <none> and password file <none>
I20260923 15:15:21.895138 8830 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:21.895177 8830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:21.895263 8830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:21.896232 8830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "fbbcc64d25aa41bb906f0884600ecda9"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "e5ba4948c444b534ed11bec5572f0382"
server_key_iv: "6d76b5e96bf084305d5bccccde76fee0"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.896522 8830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "fbbcc64d25aa41bb906f0884600ecda9"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "e5ba4948c444b534ed11bec5572f0382"
server_key_iv: "6d76b5e96bf084305d5bccccde76fee0"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.897718 8830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:21.898455 8844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.898626 8830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.898713 8830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "fbbcc64d25aa41bb906f0884600ecda9"
format_stamp: "Formatted at 2026-09-23 15:15:21 on dist-test-slave-t5lt"
server_key: "e5ba4948c444b534ed11bec5572f0382"
server_key_iv: "6d76b5e96bf084305d5bccccde76fee0"
server_key_version: "encryptionkey@0"
I20260923 15:15:21.898790 8830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:21.934947 8830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:21.935313 8830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:21.935457 8830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:21.935732 8830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:21.936074 8830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:21.936107 8830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.936151 8830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:21.936179 8830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:21.941843 8830 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:40343
I20260923 15:15:21.941926 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:40343 every 8 connection(s)
I20260923 15:15:21.942289 8830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:15:21.944368 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 8830
I20260923 15:15:21.944491 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:15:21.944693 22321 external_mini_cluster.cc:1428] Setting key cf906362ee6e9f1ec73b94ef7d0529a8
I20260923 15:15:21.947798 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.943006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:52001 (local address 127.21.204.126:42769)
0923 15:15:21.943059 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.943061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.943333 (+ 272us) server_negotiation.cc:407] Connection header received
0923 15:15:21.943805 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.943806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.943817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.943872 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:21.944657 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.945383 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.946451 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.946662 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.946749 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.947512 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:21.947513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:21.947513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:21.947518 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:21.947534 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:21.947539 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:21.947588 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.947685 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.947714 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 15:15:21.948025 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.943033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42769 (local address 127.21.204.67:52001)
0923 15:15:21.943205 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.943240 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.943719 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.943925 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.943928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.943937 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:21.944564 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.944572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.945482 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.945483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.946349 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.946352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.946763 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.946764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.946901 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.947363 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:21.947364 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:21.947392 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:21.947414 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:21.947595 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:21.947600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:21.947603 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.947686 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.947805 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260923 15:15:21.948532 8958 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42769
I20260923 15:15:21.948625 8958 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:21.948801 8958 heartbeater.cc:511] Master 127.21.204.126:42769 requested a full tablet report, sending...
I20260923 15:15:21.949185 8505 ts_manager.cc:194] Registered new tserver with Master: fbbcc64d25aa41bb906f0884600ecda9 (127.21.204.67:40343)
I20260923 15:15:21.949554 8505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.67:52001
I20260923 15:15:21.956480 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:15:21.960947 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.957797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.21.204.126:42769)
0923 15:15:21.957860 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.957862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.958047 (+ 185us) server_negotiation.cc:407] Connection header received
0923 15:15:21.958148 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.958149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.958158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.958202 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:21.958439 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.958912 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.959692 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.959888 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.959942 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.960614 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:21.960615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:21.960615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:21.960620 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:21.960633 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:21.960636 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:21.960723 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.960818 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.960828 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 15:15:21.962587 8505 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:15:21.964159 8505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:15:21.970597 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.966319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:41203 (local address 127.0.0.1:55920)
0923 15:15:21.966632 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.966666 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.966760 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.967019 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.967022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.967024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.967096 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.967099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.968425 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.968426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.969343 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.969345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.970264 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.970265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.970363 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.970378 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.970463 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.970528 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260923 15:15:21.970602 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.966416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.21.204.65:41203)
0923 15:15:21.966608 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.966613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.966742 (+ 129us) server_negotiation.cc:407] Connection header received
0923 15:15:21.966851 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.966853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.966868 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.966939 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.967181 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.968323 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.969429 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.970180 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.970259 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.970365 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.970481 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.970519 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":123,"threads_started":1}
I20260923 15:15:21.970858 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.966526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:39471 (local address 127.0.0.1:33904)
0923 15:15:21.966883 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.966893 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.967109 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.967304 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.967305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.967307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.967357 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.967359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.968182 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.968183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.969449 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.969452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.970360 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.970361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.970412 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.970415 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.970468 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.970793 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":269,"thread_start_us":65,"threads_started":1}
I20260923 15:15:21.970647 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.966630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.21.204.66:39471)
0923 15:15:21.967026 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.967030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.967041 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:15:21.967210 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.967213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.967229 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.967307 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.967422 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.968093 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.969541 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.970286 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.970352 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.970426 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.970516 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.970565 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":95,"threads_started":1}
I20260923 15:15:21.971393 8627 tablet_service.cc:1467] Processing CreateTablet for tablet 81add390b7ca42cf9c3760b065f90a16 (DEFAULT_TABLE table=test-table [id=8d79871bc0d14116a7f9b794c1bee714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:21.971540 8759 tablet_service.cc:1467] Processing CreateTablet for tablet 81add390b7ca42cf9c3760b065f90a16 (DEFAULT_TABLE table=test-table [id=8d79871bc0d14116a7f9b794c1bee714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:21.971776 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81add390b7ca42cf9c3760b065f90a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:21.971822 8759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81add390b7ca42cf9c3760b065f90a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:21.975220 8973 tablet_bootstrap.cc:492] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Bootstrap starting.
I20260923 15:15:21.975407 8972 tablet_bootstrap.cc:492] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd: Bootstrap starting.
I20260923 15:15:21.976002 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.966324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:40343 (local address 127.0.0.1:40874)
0923 15:15:21.966817 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:21.966828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:21.966898 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:21.967111 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:21.967112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:21.967114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:21.967149 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.967151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.968407 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.968408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.969465 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:21.969466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.975761 (+ 6295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.975763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:21.975827 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.975831 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:21.975887 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 15:15:21.975932 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":55,"threads_started":1}
I20260923 15:15:21.976003 8973 tablet_bootstrap.cc:654] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:21.976015 8972 tablet_bootstrap.cc:654] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:21.976013 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:21.966655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.21.204.67:40343)
0923 15:15:21.966836 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:21.966841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:21.966897 (+ 56us) server_negotiation.cc:407] Connection header received
0923 15:15:21.966963 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:21.966965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:21.966979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:21.967053 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:21.967224 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.968328 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.970474 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:21.975660 (+ 5186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:21.975758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:21.975852 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:21.975901 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 15:15:21.975936 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260923 15:15:21.976342 8973 log.cc:824] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:21.976676 8892 tablet_service.cc:1467] Processing CreateTablet for tablet 81add390b7ca42cf9c3760b065f90a16 (DEFAULT_TABLE table=test-table [id=8d79871bc0d14116a7f9b794c1bee714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:21.977026 8892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81add390b7ca42cf9c3760b065f90a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:21.977097 8973 tablet_bootstrap.cc:492] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: No bootstrap required, opened a new log
I20260923 15:15:21.977160 8973 ts_tablet_manager.cc:1397] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:15:21.979866 8973 raft_consensus.cc:348] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:21.980041 8973 raft_consensus.cc:374] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:21.980088 8973 raft_consensus.cc:729] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123d2085bc934f8d989807df0c7105ac, State: Initialized, Role: FOLLOWER
I20260923 15:15:21.980216 8973 consensus_queue.cc:260] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:21.980487 8973 ts_tablet_manager.cc:1428] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 15:15:21.980612 8826 heartbeater.cc:503] Master 127.21.204.126:42769 was elected leader, sending a full tablet report...
I20260923 15:15:21.980592 8976 tablet_bootstrap.cc:492] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Bootstrap starting.
I20260923 15:15:21.980661 8972 log.cc:824] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:21.981343 8976 tablet_bootstrap.cc:654] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:21.981549 8972 tablet_bootstrap.cc:492] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd: No bootstrap required, opened a new log
I20260923 15:15:21.981617 8972 ts_tablet_manager.cc:1397] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260923 15:15:21.981680 8976 log.cc:824] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:21.982595 8976 tablet_bootstrap.cc:492] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: No bootstrap required, opened a new log
I20260923 15:15:21.982647 8976 ts_tablet_manager.cc:1397] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 15:15:21.983700 8972 raft_consensus.cc:348] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:21.983847 8972 raft_consensus.cc:374] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:21.983894 8972 raft_consensus.cc:729] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18c041befb243a19d689dfe3e576edd, State: Initialized, Role: FOLLOWER
I20260923 15:15:21.984005 8972 consensus_queue.cc:260] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:21.984308 8972 ts_tablet_manager.cc:1428] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:15:21.984354 8693 heartbeater.cc:503] Master 127.21.204.126:42769 was elected leader, sending a full tablet report...
I20260923 15:15:21.984704 8976 raft_consensus.cc:348] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:21.984870 8976 raft_consensus.cc:374] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:21.984907 8976 raft_consensus.cc:729] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbcc64d25aa41bb906f0884600ecda9, State: Initialized, Role: FOLLOWER
I20260923 15:15:21.985009 8976 consensus_queue.cc:260] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:21.985252 8976 ts_tablet_manager.cc:1428] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 15:15:21.985288 8958 heartbeater.cc:503] Master 127.21.204.126:42769 was elected leader, sending a full tablet report...
W20260923 15:15:22.040452 8828 tablet.cc:2369] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:15:22.103606 8979 raft_consensus.cc:482] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:15:22.103754 8979 raft_consensus.cc:504] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:22.104182 8979 leader_election.cc:288] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbcc64d25aa41bb906f0884600ecda9 (127.21.204.67:40343), 123d2085bc934f8d989807df0c7105ac (127.21.204.66:39471)
I20260923 15:15:22.108217 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:22.104644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:35283 (local address 127.21.204.67:40343)
0923 15:15:22.104701 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:22.104704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:22.104713 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:22.104759 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:22.104759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:22.104769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:22.104827 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:22.105032 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.105800 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.107291 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.107846 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.107924 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:22.107933 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:22.108101 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 15:15:22.108142 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 15:15:22.108217 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:22.104655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:54357 (local address 127.21.204.66:39471)
0923 15:15:22.104705 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:22.104707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:22.104795 (+ 88us) server_negotiation.cc:407] Connection header received
0923 15:15:22.104861 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:22.104862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:22.104874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:22.104929 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:22.105097 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.105797 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.107291 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.107846 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.107924 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:22.107933 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:22.108101 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 15:15:22.108142 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 15:15:22.108227 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:22.104436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:40343 (local address 127.21.204.65:35283)
0923 15:15:22.104517 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:22.104531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:22.104614 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:22.104854 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:22.104855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:22.104857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:22.104954 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:22.104956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.105907 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.105908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:22.107196 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:22.107199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.107939 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.107940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:22.108012 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:22.108019 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 15:15:22.108077 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 15:15:22.108128 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":41}
I20260923 15:15:22.108232 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:22.104553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:39471 (local address 127.21.204.65:54357)
0923 15:15:22.104725 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:22.104733 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:22.104785 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:22.104960 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:22.104961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:22.104962 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:22.105023 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:22.105024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.105907 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.105916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:22.107196 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:22.107199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.107955 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.107955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:22.108012 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:22.108020 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 15:15:22.108077 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 15:15:22.108128 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":54,"thread_start_us":101,"threads_started":1}
I20260923 15:15:22.108639 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81add390b7ca42cf9c3760b065f90a16" candidate_uuid: "f18c041befb243a19d689dfe3e576edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbcc64d25aa41bb906f0884600ecda9" is_pre_election: true
I20260923 15:15:22.108762 8912 raft_consensus.cc:2454] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f18c041befb243a19d689dfe3e576edd in term 0.
I20260923 15:15:22.108989 8580 leader_election.cc:302] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f18c041befb243a19d689dfe3e576edd, fbbcc64d25aa41bb906f0884600ecda9; no voters:
I20260923 15:15:22.109094 8780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81add390b7ca42cf9c3760b065f90a16" candidate_uuid: "f18c041befb243a19d689dfe3e576edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123d2085bc934f8d989807df0c7105ac" is_pre_election: true
I20260923 15:15:22.109138 8979 raft_consensus.cc:2781] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:15:22.109210 8979 raft_consensus.cc:482] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:15:22.109246 8979 raft_consensus.cc:3037] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:22.109272 8780 raft_consensus.cc:2454] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f18c041befb243a19d689dfe3e576edd in term 0.
I20260923 15:15:22.110064 8979 raft_consensus.cc:504] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:22.110211 8979 leader_election.cc:288] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [CANDIDATE]: Term 1 election: Requested vote from peers fbbcc64d25aa41bb906f0884600ecda9 (127.21.204.67:40343), 123d2085bc934f8d989807df0c7105ac (127.21.204.66:39471)
I20260923 15:15:22.110358 8912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81add390b7ca42cf9c3760b065f90a16" candidate_uuid: "f18c041befb243a19d689dfe3e576edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbcc64d25aa41bb906f0884600ecda9"
I20260923 15:15:22.110410 8780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81add390b7ca42cf9c3760b065f90a16" candidate_uuid: "f18c041befb243a19d689dfe3e576edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "123d2085bc934f8d989807df0c7105ac"
I20260923 15:15:22.110453 8912 raft_consensus.cc:3037] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:22.110469 8780 raft_consensus.cc:3037] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:22.111356 8780 raft_consensus.cc:2454] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f18c041befb243a19d689dfe3e576edd in term 1.
I20260923 15:15:22.111356 8912 raft_consensus.cc:2454] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f18c041befb243a19d689dfe3e576edd in term 1.
I20260923 15:15:22.111548 8580 leader_election.cc:302] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123d2085bc934f8d989807df0c7105ac, f18c041befb243a19d689dfe3e576edd; no voters:
I20260923 15:15:22.111657 8979 raft_consensus.cc:2781] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:22.111814 8979 raft_consensus.cc:686] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [term 1 LEADER]: Becoming Leader. State: Replica: f18c041befb243a19d689dfe3e576edd, State: Running, Role: LEADER
I20260923 15:15:22.111919 8979 consensus_queue.cc:237] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } }
I20260923 15:15:22.112777 8505 catalog_manager.cc:5495] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd reported cstate change: term changed from 0 to 1, leader changed from <none> to f18c041befb243a19d689dfe3e576edd (127.21.204.65). New cstate: current_term: 1 leader_uuid: "f18c041befb243a19d689dfe3e576edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f18c041befb243a19d689dfe3e576edd" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 } health_report { overall_health: UNKNOWN } } }
I20260923 15:15:22.151441 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:22.148776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.21.204.65:41203)
0923 15:15:22.148823 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:22.148827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:22.148839 (+ 12us) server_negotiation.cc:407] Connection header received
0923 15:15:22.148875 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:22.148876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:22.148895 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:22.148975 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:22.149130 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.149645 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.150607 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:22.150851 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:22.150932 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:22.151075 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:22.151173 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:22.151284 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:22.151322 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 15:15:22.151331 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:15:22.153409 8780 raft_consensus.cc:1261] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 1 FOLLOWER]: Refusing update from remote peer f18c041befb243a19d689dfe3e576edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:22.153409 8912 raft_consensus.cc:1261] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 1 FOLLOWER]: Refusing update from remote peer f18c041befb243a19d689dfe3e576edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:22.153797 8982 consensus_queue.cc:1035] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbcc64d25aa41bb906f0884600ecda9" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:22.153971 8979 consensus_queue.cc:1035] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "123d2085bc934f8d989807df0c7105ac" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:22.156214 8985 mvcc.cc:205] Tried to move back new op lower bound from 7332563034737053696 to 7332563034571120640. Current Snapshot: MvccSnapshot[applied={T|T < 7332563034737053696}]
I20260923 15:15:22.157634 8986 mvcc.cc:205] Tried to move back new op lower bound from 7332563034737053696 to 7332563034571120640. Current Snapshot: MvccSnapshot[applied={T|T < 7332563034737053696 or (T in {7332563034737053696})}]
I20260923 15:15:22.158847 8504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:22.158950 8504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:22.159729 8504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8307bb4501064dd5b8fb931ba265b2ad: Sending DeleteTablet for 3 replicas of tablet 81add390b7ca42cf9c3760b065f90a16
I20260923 15:15:22.160068 8627 tablet_service.cc:1514] Processing DeleteTablet for tablet 81add390b7ca42cf9c3760b065f90a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:22 UTC) from {username='slave'} at 127.0.0.1:55920
I20260923 15:15:22.160140 8892 tablet_service.cc:1514] Processing DeleteTablet for tablet 81add390b7ca42cf9c3760b065f90a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:22 UTC) from {username='slave'} at 127.0.0.1:40874
I20260923 15:15:22.160149 8759 tablet_service.cc:1514] Processing DeleteTablet for tablet 81add390b7ca42cf9c3760b065f90a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:22 UTC) from {username='slave'} at 127.0.0.1:33904
I20260923 15:15:22.160488 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 8565
I20260923 15:15:22.160611 8995 tablet_replica.cc:330] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: stopping tablet replica
I20260923 15:15:22.160686 8994 tablet_replica.cc:330] T 81add390b7ca42cf9c3760b065f90a16 P f18c041befb243a19d689dfe3e576edd: stopping tablet replica
I20260923 15:15:22.161141 8995 raft_consensus.cc:2229] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:22.161267 8995 raft_consensus.cc:2258] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:22.161448 8996 tablet_replica.cc:330] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: stopping tablet replica
I20260923 15:15:22.161541 8996 raft_consensus.cc:2229] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:22.161638 8996 raft_consensus.cc:2258] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:22.161859 8995 ts_tablet_manager.cc:1905] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:22.162784 8996 ts_tablet_manager.cc:1905] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:22.163717 8995 ts_tablet_manager.cc:1918] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:22.163796 8995 log.cc:1196] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/81add390b7ca42cf9c3760b065f90a16
I20260923 15:15:22.164037 8995 ts_tablet_manager.cc:1939] T 81add390b7ca42cf9c3760b065f90a16 P fbbcc64d25aa41bb906f0884600ecda9: Deleting consensus metadata
I20260923 15:15:22.164402 8996 ts_tablet_manager.cc:1918] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:22.164470 8996 log.cc:1196] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/81add390b7ca42cf9c3760b065f90a16
I20260923 15:15:22.164556 8489 catalog_manager.cc:4841] TS fbbcc64d25aa41bb906f0884600ecda9 (127.21.204.67:40343): tablet 81add390b7ca42cf9c3760b065f90a16 (table test-table [id=8d79871bc0d14116a7f9b794c1bee714]) successfully deleted
I20260923 15:15:22.164757 8996 ts_tablet_manager.cc:1939] T 81add390b7ca42cf9c3760b065f90a16 P 123d2085bc934f8d989807df0c7105ac: Deleting consensus metadata
I20260923 15:15:22.165138 8489 catalog_manager.cc:4841] TS 123d2085bc934f8d989807df0c7105ac (127.21.204.66:39471): tablet 81add390b7ca42cf9c3760b065f90a16 (table test-table [id=8d79871bc0d14116a7f9b794c1bee714]) successfully deleted
W20260923 15:15:22.166689 8492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:41203 (error 108)
I20260923 15:15:22.167135 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:22.166999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:41203 (local address 127.0.0.1:55942)
0923 15:15:22.167056 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:22.167097 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 15:15:22.167171 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 8697
W20260923 15:15:22.167249 8492 catalog_manager.cc:4584] TS f18c041befb243a19d689dfe3e576edd (127.21.204.65:41203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81add390b7ca42cf9c3760b065f90a16: Network error: Client connection negotiation failed: client connection to 127.21.204.65:41203: connect: Connection refused (error 111)
I20260923 15:15:22.172421 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 8830
I20260923 15:15:22.176854 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 8474
2026-09-23T15:15:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (861 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:22 dist-test-slave-t5lt krb5kdc[9000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:22 dist-test-slave-t5lt krb5kdc[9000](info): set up 2 sockets
Sep 23 15:15:22 dist-test-slave-t5lt krb5kdc[9000](info): commencing operation
krb5kdc: starting...
W20260923 15:15:23.210978 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:23.328174 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42103
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:42103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:23.409822 9013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:23.410069 9013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:23.410094 9013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:23.411177 9013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:23.411217 9013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:23.411233 9013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:23.411247 9013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:23.411262 9013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:23.412667 9013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:42103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:23.412870 9013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:23.413048 9013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:23.418157 9019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.418193 9021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.418157 9018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:23.418560 9013 server_base.cc:1034] running on GCE node
I20260923 15:15:23.418792 9013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:23.419025 9013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:23.420279 9013 hybrid_clock.cc:648] HybridClock initialized: now 1790176523420267 us; error 42 us; skew 500 ppm
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:23.422865 9013 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:23.423373 9013 webserver.cc:460] Webserver started at http://127.21.204.126:37099/ using document root <none> and password file <none>
I20260923 15:15:23.423583 9013 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:23.423625 9013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:23.423712 9013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:23.424667 9013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "34033960497a461b9a239d4ddd9a3d9e"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "d79cc9c69beae7acd028d75d1fbd8358"
server_key_iv: "6f679aadd4f3e86c3cb2821f815296df"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.424958 9013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "34033960497a461b9a239d4ddd9a3d9e"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "d79cc9c69beae7acd028d75d1fbd8358"
server_key_iv: "6f679aadd4f3e86c3cb2821f815296df"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.426209 9013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:23.426909 9028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.427098 9013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:15:23.427166 9013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "34033960497a461b9a239d4ddd9a3d9e"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "d79cc9c69beae7acd028d75d1fbd8358"
server_key_iv: "6f679aadd4f3e86c3cb2821f815296df"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.427217 9013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:23.438849 9013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:23.440634 9013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:23.440743 9013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:23.445009 9013 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:42103
I20260923 15:15:23.445088 9080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:42103 every 8 connection(s)
I20260923 15:15:23.445410 9013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:23.446367 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:23.448562 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e: Bootstrap starting.
I20260923 15:15:23.449133 9081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:23.449424 9081 log.cc:824] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:23.450170 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e: No bootstrap required, opened a new log
I20260923 15:15:23.451722 9081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34033960497a461b9a239d4ddd9a3d9e" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42103 } }
I20260923 15:15:23.451865 9081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:23.451889 9081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34033960497a461b9a239d4ddd9a3d9e, State: Initialized, Role: FOLLOWER
I20260923 15:15:23.452042 9081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34033960497a461b9a239d4ddd9a3d9e" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42103 } }
I20260923 15:15:23.452105 9081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:23.452126 9081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:23.452176 9081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:23.452716 9081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34033960497a461b9a239d4ddd9a3d9e" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42103 } }
I20260923 15:15:23.452839 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9013
I20260923 15:15:23.452847 9081 leader_election.cc:302] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34033960497a461b9a239d4ddd9a3d9e; no voters:
I20260923 15:15:23.452960 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:23.453013 9081 leader_election.cc:288] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:23.453095 9084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:23.453183 22321 external_mini_cluster.cc:1428] Setting key fdb6e3ecb1c0cd86fa02fd773597a972
I20260923 15:15:23.453245 9084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [term 1 LEADER]: Becoming Leader. State: Replica: 34033960497a461b9a239d4ddd9a3d9e, State: Running, Role: LEADER
I20260923 15:15:23.453265 9081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:23.453344 9084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34033960497a461b9a239d4ddd9a3d9e" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42103 } }
I20260923 15:15:23.453629 9084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34033960497a461b9a239d4ddd9a3d9e. Latest consensus state: current_term: 1 leader_uuid: "34033960497a461b9a239d4ddd9a3d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34033960497a461b9a239d4ddd9a3d9e" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42103 } } }
I20260923 15:15:23.453711 9084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:23.453862 9086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34033960497a461b9a239d4ddd9a3d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34033960497a461b9a239d4ddd9a3d9e" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 42103 } } }
I20260923 15:15:23.454010 9086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:23.454610 9091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:23.455140 9091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:23.456682 9091 catalog_manager.cc:1347] Generated new cluster ID: 900bdb6b9e044e6bb35f815f10f5ed05
I20260923 15:15:23.456736 9091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:23.464268 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.453801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.21.204.126:42103)
0923 15:15:23.454363 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.454374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.454412 (+ 38us) server_negotiation.cc:407] Connection header received
0923 15:15:23.454536 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.454545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.454604 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.455504 (+ 900us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:23.455579 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.456810 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.457582 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.458088 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.458215 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.460459 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:23.460463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:23.460463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:23.460475 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:23.462659 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.463025 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.463028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.463028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.463127 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.463346 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.463347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.463347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.463582 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:23.463744 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.463853 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.464107 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":102,"threads_started":1}
I20260923 15:15:23.480322 9091 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:23.480849 9091 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:23.483548 9091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e: Generated new TSK 0
I20260923 15:15:23.483718 9091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.204.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
Entry for principal kudu/127.21.204.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab.
I20260923 15:15:23.514353 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42103
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:23.594900 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:23.595088 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:23.595110 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:23.596163 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:23.596194 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:23.596220 9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:15:23.597663 9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.65.keytab
--principal=kudu/127.21.204.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:23.597853 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:23.598086 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:23.603019 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.603029 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.603029 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:23.603305 9107 server_base.cc:1034] running on GCE node
I20260923 15:15:23.603493 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:23.603719 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:23.604938 9107 hybrid_clock.cc:648] HybridClock initialized: now 1790176523604921 us; error 45 us; skew 500 ppm
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:23.607345 9107 init.cc:377] Logged in from keytab as kudu/127.21.204.65@KRBTEST.COM (short username kudu)
I20260923 15:15:23.607741 9107 webserver.cc:460] Webserver started at http://127.21.204.65:45145/ using document root <none> and password file <none>
I20260923 15:15:23.607973 9107 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:23.608042 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:23.608194 9107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:23.609170 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "3b01f4fae77b4b4da1e6b0941605e68a"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "62f2a408a48a3665ad4a49e3cff34c70"
server_key_iv: "1a3c16a1f2d5874fb6de3d40cddfa5bd"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.609513 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "3b01f4fae77b4b4da1e6b0941605e68a"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "62f2a408a48a3665ad4a49e3cff34c70"
server_key_iv: "1a3c16a1f2d5874fb6de3d40cddfa5bd"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.610875 9107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:23.611594 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.611765 9107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:15:23.611862 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "3b01f4fae77b4b4da1e6b0941605e68a"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "62f2a408a48a3665ad4a49e3cff34c70"
server_key_iv: "1a3c16a1f2d5874fb6de3d40cddfa5bd"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.611949 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:23.638909 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:23.640836 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:23.641010 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:23.641245 9107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:23.641544 9107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:23.641597 9107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.641633 9107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:23.641662 9107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.647684 9107 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:41713
I20260923 15:15:23.647737 9235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:41713 every 8 connection(s)
I20260923 15:15:23.648032 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:15:23.648893 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9107
I20260923 15:15:23.649015 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:15:23.649223 22321 external_mini_cluster.cc:1428] Setting key 48d88e228ea01c4f876063c9e5d9665a
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.65@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.66@KRBTEST.COM; defaulting to no policy
I20260923 15:15:23.657781 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.648751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:36891 (local address 127.21.204.126:42103)
0923 15:15:23.648838 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.648840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.649048 (+ 208us) server_negotiation.cc:407] Connection header received
0923 15:15:23.649672 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.649673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.649682 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.649724 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:23.650705 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.651209 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.652189 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.652412 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.652492 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.655035 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:23.655036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:23.655037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:23.655043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:23.656353 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.656810 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.656811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.656811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.656861 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.657151 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.657152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.657152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.657246 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:23.657294 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.657656 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.657718 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 15:15:23.658073 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.648698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42103 (local address 127.21.204.65:36891)
0923 15:15:23.648905 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:23.648960 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:23.649542 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:23.649792 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:23.649796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:23.650411 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:23.650603 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.650608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.651306 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.651307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.652095 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.652097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.652514 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.652515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.652699 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.653231 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:23.653242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:23.654877 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:23.656437 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:23.656445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:23.656447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:23.656733 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:23.656923 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:23.656924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:23.656924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:23.657082 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:23.657314 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:23.657322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:23.657521 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 15:15:23.657636 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 15:15:23.657796 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
I20260923 15:15:23.658613 9236 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42103
I20260923 15:15:23.658694 9236 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:23.658952 9236 heartbeater.cc:511] Master 127.21.204.126:42103 requested a full tablet report, sending...
I20260923 15:15:23.659446 9045 ts_manager.cc:194] Registered new tserver with Master: 3b01f4fae77b4b4da1e6b0941605e68a (127.21.204.65:41713)
I20260923 15:15:23.660044 9045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.65@KRBTEST.COM'} at 127.21.204.65:36891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
Entry for principal kudu/127.21.204.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab.
I20260923 15:15:23.676303 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42103
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:23.757393 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:23.757584 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:23.757608 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:23.758764 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:23.758805 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:23.758831 9241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:15:23.760197 9241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.66.keytab
--principal=kudu/127.21.204.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:23.760440 9241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:23.760622 9241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:23.765630 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.765641 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.765641 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:23.765822 9241 server_base.cc:1034] running on GCE node
I20260923 15:15:23.766090 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:23.766327 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:23.767493 9241 hybrid_clock.cc:648] HybridClock initialized: now 1790176523767472 us; error 41 us; skew 500 ppm
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:23.770156 9241 init.cc:377] Logged in from keytab as kudu/127.21.204.66@KRBTEST.COM (short username kudu)
I20260923 15:15:23.770654 9241 webserver.cc:460] Webserver started at http://127.21.204.66:44559/ using document root <none> and password file <none>
I20260923 15:15:23.770915 9241 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:23.770984 9241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:23.771137 9241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:23.772171 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "b07a65ba194142d78bacf332454f37f0"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "7bd85dc681abc9bf36bd136a3cdf0216"
server_key_iv: "ae4b1f16f79d6c724c7fc00ca18cd5ff"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.772528 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "b07a65ba194142d78bacf332454f37f0"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "7bd85dc681abc9bf36bd136a3cdf0216"
server_key_iv: "ae4b1f16f79d6c724c7fc00ca18cd5ff"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.773864 9241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:15:23.774657 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.774837 9241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:15:23.774936 9241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "b07a65ba194142d78bacf332454f37f0"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "7bd85dc681abc9bf36bd136a3cdf0216"
server_key_iv: "ae4b1f16f79d6c724c7fc00ca18cd5ff"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.775022 9241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:23.782120 9241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:23.784310 9241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:23.784467 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:23.784682 9241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:23.784978 9241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:23.785032 9241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.785070 9241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:23.785099 9241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.791277 9241 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:44021
I20260923 15:15:23.791411 9369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:44021 every 8 connection(s)
I20260923 15:15:23.792524 9241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.66@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:23.800936 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9241
I20260923 15:15:23.801070 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:15:23.801251 22321 external_mini_cluster.cc:1428] Setting key 51f277ecab81e3951c97394016f5283c
I20260923 15:15:23.802640 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.793328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:56273 (local address 127.21.204.126:42103)
0923 15:15:23.793400 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.793402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.793661 (+ 259us) server_negotiation.cc:407] Connection header received
0923 15:15:23.794205 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.794206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.794216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.794269 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:23.795178 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.795682 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.796873 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.797064 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.797120 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.799521 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:23.799523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:23.799523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:23.799532 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:23.800839 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.801414 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.801415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.801416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.801487 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.801797 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.801798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.801799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.802021 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:23.802102 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.802442 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.802537 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 15:15:23.802870 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.793226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42103 (local address 127.21.204.66:56273)
0923 15:15:23.793510 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:23.793558 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:23.794111 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:23.794320 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:23.794323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:23.794904 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:23.795095 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.795099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.795801 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.795802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.796779 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.796783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.797148 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.797149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.797288 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.797759 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:23.797768 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:23.799384 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:23.800955 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:23.800964 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:23.800967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:23.801306 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:23.801564 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:23.801565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:23.801566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:23.801703 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:23.802145 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:23.802152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:23.802285 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 15:15:23.802416 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 15:15:23.802593 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20260923 15:15:23.803413 9370 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42103
I20260923 15:15:23.803543 9370 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:23.803750 9370 heartbeater.cc:511] Master 127.21.204.126:42103 requested a full tablet report, sending...
I20260923 15:15:23.804209 9045 ts_manager.cc:194] Registered new tserver with Master: b07a65ba194142d78bacf332454f37f0 (127.21.204.66:44021)
I20260923 15:15:23.804731 9045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.66@KRBTEST.COM'} at 127.21.204.66:56273
WARNING: no policy specified for kudu/127.21.204.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
Entry for principal kudu/127.21.204.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab.
I20260923 15:15:23.827060 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42103
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:23.907388 9375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:23.907590 9375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:23.907614 9375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:23.908782 9375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:23.908829 9375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:23.908857 9375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:15:23.910853 9375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.67.keytab
--principal=kudu/127.21.204.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:42103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:23.911127 9375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:23.911425 9375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:23.918660 9380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.918697 9381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:23.918808 9383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:23.919057 9375 server_base.cc:1034] running on GCE node
I20260923 15:15:23.919302 9375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:23.919588 9375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:23.920775 9375 hybrid_clock.cc:648] HybridClock initialized: now 1790176523920760 us; error 24 us; skew 500 ppm
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:23.923316 9375 init.cc:377] Logged in from keytab as kudu/127.21.204.67@KRBTEST.COM (short username kudu)
I20260923 15:15:23.923696 9375 webserver.cc:460] Webserver started at http://127.21.204.67:38297/ using document root <none> and password file <none>
I20260923 15:15:23.923887 9375 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:23.923928 9375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:23.924013 9375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:23.925086 9375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "475b1ef5c7254855ad78fe2c6ad9b81d"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "a96c723aeb6be1bd16deb9078585c580"
server_key_iv: "f9121fccc450410a1d6e966ab8accf53"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.925383 9375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "475b1ef5c7254855ad78fe2c6ad9b81d"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "a96c723aeb6be1bd16deb9078585c580"
server_key_iv: "f9121fccc450410a1d6e966ab8accf53"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.926707 9375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:23.927351 9390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.927508 9375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:15:23.927595 9375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "475b1ef5c7254855ad78fe2c6ad9b81d"
format_stamp: "Formatted at 2026-09-23 15:15:23 on dist-test-slave-t5lt"
server_key: "a96c723aeb6be1bd16deb9078585c580"
server_key_iv: "f9121fccc450410a1d6e966ab8accf53"
server_key_version: "encryptionkey@0"
I20260923 15:15:23.927649 9375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:23.943159 9375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:23.945093 9375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:23.945288 9375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:23.945508 9375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:23.945801 9375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:23.945856 9375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.945905 9375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:23.945936 9375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:23.952845 9375 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:35505
I20260923 15:15:23.952911 9503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:35505 every 8 connection(s)
I20260923 15:15:23.953275 9375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
Sep 23 15:15:23 dist-test-slave-t5lt krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.67@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:23.962028 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9375
I20260923 15:15:23.962162 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:15:23.962343 22321 external_mini_cluster.cc:1428] Setting key 83465810c141cb973cf4932dafafefaa
I20260923 15:15:23.963605 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.954117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:60129 (local address 127.21.204.126:42103)
0923 15:15:23.954159 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.954162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.954452 (+ 290us) server_negotiation.cc:407] Connection header received
0923 15:15:23.955093 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.955094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.955108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.955167 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:23.956085 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.956593 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.957593 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.957804 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.957878 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.960220 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:23.960221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:23.960222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:23.960231 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:23.961771 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.962400 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.962402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.962402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.962471 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.962802 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.962803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.962803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.962951 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:23.963023 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.963404 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.963535 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:15:23.963703 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.954103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:42103 (local address 127.21.204.67:60129)
0923 15:15:23.954333 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:23.954369 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:23.955001 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:23.955233 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:23.955237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:23.955806 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:23.955985 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.955989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.956690 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.956691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.957491 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.957493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.957906 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.957907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.958040 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.958563 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:23.958571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:23.960110 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:23.961985 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:23.961995 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:23.961997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:23.962315 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:23.962550 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:23.962551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:23.962551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:23.962697 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:23.963025 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:23.963033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:23.963206 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 15:15:23.963342 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 15:15:23.963519 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":142,"threads_started":1}
I20260923 15:15:23.964195 9504 heartbeater.cc:347] Connected to a master server at 127.21.204.126:42103
I20260923 15:15:23.964298 9504 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:23.964502 9504 heartbeater.cc:511] Master 127.21.204.126:42103 requested a full tablet report, sending...
I20260923 15:15:23.964910 9045 ts_manager.cc:194] Registered new tserver with Master: 475b1ef5c7254855ad78fe2c6ad9b81d (127.21.204.67:35505)
I20260923 15:15:23.965356 9045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.204.67@KRBTEST.COM'} at 127.21.204.67:60129
I20260923 15:15:23.974494 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:15:23.982010 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.975976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.21.204.126:42103)
0923 15:15:23.976059 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.976061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.976111 (+ 50us) server_negotiation.cc:407] Connection header received
0923 15:15:23.976263 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.976264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.976274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.976336 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:23.976867 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.977328 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.978120 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.978305 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.978368 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.979624 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:23.979625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:23.979626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:23.979632 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:23.980879 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.981202 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.981203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.981203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.981254 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:23.981471 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:23.981472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:23.981472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:23.981609 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:23.981685 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.981796 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.981860 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 15:15:23.983732 9045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:15:23.985378 9045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:15:23.991501 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.987638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:35505 (local address 127.0.0.1:54278)
0923 15:15:23.987786 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:23.987842 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:23.987903 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:23.988141 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:23.988144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:23.988146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:23.988213 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.988215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.989491 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.989492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.990410 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.990413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.991199 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.991200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.991279 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.991296 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 15:15:23.991376 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 15:15:23.991427 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1,"thread_start_us":42,"threads_started":1}
I20260923 15:15:23.991529 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.987736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.21.204.67:35505)
0923 15:15:23.987960 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.987968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.987978 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:15:23.988019 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.988021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.988038 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.988114 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:23.988285 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.989380 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.990503 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.991112 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.991187 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.991260 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.991405 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.991447 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260923 15:15:23.991945 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.987638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:41713 (local address 127.0.0.1:37710)
0923 15:15:23.987937 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:23.987948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:23.988006 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:23.988449 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:23.988450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:23.988452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:23.988490 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.988491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.989551 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.989552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.990725 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.990728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.991714 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.991714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.991762 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.991765 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:23.991840 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 15:15:23.991880 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":30,"thread_start_us":75,"threads_started":1}
I20260923 15:15:23.991981 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.987767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.21.204.65:41713)
0923 15:15:23.988307 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.988312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.988321 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:23.988361 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.988363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.988380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.988451 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:23.988605 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.989435 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.990826 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.991546 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.991619 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.991682 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.991857 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.991902 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":77,"threads_started":1}
I20260923 15:15:23.992522 9170 tablet_service.cc:1467] Processing CreateTablet for tablet 6a91cb3770b64b5491080b4e95bc1656 (DEFAULT_TABLE table=test-table [id=9f538d076dd144d1b8a7fa03e64394ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:23.992509 9438 tablet_service.cc:1467] Processing CreateTablet for tablet 6a91cb3770b64b5491080b4e95bc1656 (DEFAULT_TABLE table=test-table [id=9f538d076dd144d1b8a7fa03e64394ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:23.992785 9170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a91cb3770b64b5491080b4e95bc1656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:23.992851 9438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a91cb3770b64b5491080b4e95bc1656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:23.995337 9521 tablet_bootstrap.cc:492] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a: Bootstrap starting.
I20260923 15:15:23.995680 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.987810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:44021 (local address 127.0.0.1:50696)
0923 15:15:23.988566 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:23.988576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:23.988626 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:23.988803 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:23.988803 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:23.988805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:23.988850 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.988851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.990091 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.990092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.991255 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:23.991257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.992166 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.992167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:23.995481 (+ 3314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.995488 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 15:15:23.995543 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 15:15:23.995589 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260923 15:15:23.995956 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:23.988041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.21.204.66:44021)
0923 15:15:23.988240 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:23.988244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:23.988626 (+ 382us) server_negotiation.cc:407] Connection header received
0923 15:15:23.988696 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:23.988698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:23.988714 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:23.988790 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:23.988920 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.989980 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.991346 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:23.992085 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:23.992164 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:23.992217 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:23.995804 (+ 3587us) server_negotiation.cc:299] Negotiation successful
0923 15:15:23.995853 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
I20260923 15:15:23.996022 9521 tablet_bootstrap.cc:654] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:23.996608 9521 log.cc:824] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:23.996702 9304 tablet_service.cc:1467] Processing CreateTablet for tablet 6a91cb3770b64b5491080b4e95bc1656 (DEFAULT_TABLE table=test-table [id=9f538d076dd144d1b8a7fa03e64394ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:23.996954 9522 tablet_bootstrap.cc:492] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Bootstrap starting.
I20260923 15:15:23.997040 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a91cb3770b64b5491080b4e95bc1656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:23.997423 9521 tablet_bootstrap.cc:492] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a: No bootstrap required, opened a new log
I20260923 15:15:23.997511 9521 ts_tablet_manager.cc:1397] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:23.997550 9522 tablet_bootstrap.cc:654] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:23.998127 9522 log.cc:824] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:23.998920 9522 tablet_bootstrap.cc:492] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: No bootstrap required, opened a new log
I20260923 15:15:23.998986 9522 ts_tablet_manager.cc:1397] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:15:23.999406 9521 raft_consensus.cc:348] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:23.999572 9521 raft_consensus.cc:374] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:23.999615 9521 raft_consensus.cc:729] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b01f4fae77b4b4da1e6b0941605e68a, State: Initialized, Role: FOLLOWER
I20260923 15:15:23.999732 9521 consensus_queue.cc:260] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.000000 9521 ts_tablet_manager.cc:1428] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 15:15:24.000142 9236 heartbeater.cc:503] Master 127.21.204.126:42103 was elected leader, sending a full tablet report...
I20260923 15:15:24.000427 9526 tablet_bootstrap.cc:492] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Bootstrap starting.
I20260923 15:15:24.001066 9526 tablet_bootstrap.cc:654] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:24.001466 9522 raft_consensus.cc:348] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.001646 9522 raft_consensus.cc:374] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:24.001655 9526 log.cc:824] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:24.001698 9522 raft_consensus.cc:729] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475b1ef5c7254855ad78fe2c6ad9b81d, State: Initialized, Role: FOLLOWER
I20260923 15:15:24.001811 9522 consensus_queue.cc:260] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.002115 9522 ts_tablet_manager.cc:1428] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260923 15:15:24.002144 9504 heartbeater.cc:503] Master 127.21.204.126:42103 was elected leader, sending a full tablet report...
I20260923 15:15:24.002674 9526 tablet_bootstrap.cc:492] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: No bootstrap required, opened a new log
I20260923 15:15:24.002740 9526 ts_tablet_manager.cc:1397] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:24.004565 9526 raft_consensus.cc:348] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.004699 9526 raft_consensus.cc:374] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:24.004720 9526 raft_consensus.cc:729] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07a65ba194142d78bacf332454f37f0, State: Initialized, Role: FOLLOWER
I20260923 15:15:24.004835 9526 consensus_queue.cc:260] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.005084 9526 ts_tablet_manager.cc:1428] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 15:15:24.005141 9370 heartbeater.cc:503] Master 127.21.204.126:42103 was elected leader, sending a full tablet report...
W20260923 15:15:24.043170 9371 tablet.cc:2369] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:15:24.070261 9529 raft_consensus.cc:482] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:15:24.070458 9529 raft_consensus.cc:504] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.070878 9529 leader_election.cc:288] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 475b1ef5c7254855ad78fe2c6ad9b81d (127.21.204.67:35505), 3b01f4fae77b4b4da1e6b0941605e68a (127.21.204.65:41713)
I20260923 15:15:24.073693 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.071224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:39893 (local address 127.21.204.67:35505)
0923 15:15:24.071276 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.071278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.071286 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:15:24.071314 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.071315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.071336 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.071385 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:24.071546 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.072031 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.072954 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.073339 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.073424 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.073435 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.073587 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.073624 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:24.073727 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.071052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:35505 (local address 127.21.204.66:39893)
0923 15:15:24.071118 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.071132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.071182 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.071388 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.071389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.071390 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:24.071472 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.071473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.072120 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.072121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.072850 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.072852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.073424 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.073425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.073508 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.073511 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.073580 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.073623 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 15:15:24.073755 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.071167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:54997 (local address 127.21.204.65:41713)
0923 15:15:24.071215 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.071217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.071358 (+ 141us) server_negotiation.cc:407] Connection header received
0923 15:15:24.071385 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.071386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.071397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.071465 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:24.071631 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.072098 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.073022 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.073404 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.073489 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.073497 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.073650 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.073684 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:15:24.073848 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.071065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:41713 (local address 127.21.204.66:54997)
0923 15:15:24.071247 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.071257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.071312 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.071473 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.071475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.071477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:24.071554 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.071556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.072183 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.072184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.072928 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.072930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.073490 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.073491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.073567 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.073570 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.073631 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.073670 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260923 15:15:24.074208 9190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a91cb3770b64b5491080b4e95bc1656" candidate_uuid: "b07a65ba194142d78bacf332454f37f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" is_pre_election: true
I20260923 15:15:24.074419 9190 raft_consensus.cc:2454] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b07a65ba194142d78bacf332454f37f0 in term 0.
I20260923 15:15:24.074426 9458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a91cb3770b64b5491080b4e95bc1656" candidate_uuid: "b07a65ba194142d78bacf332454f37f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" is_pre_election: true
I20260923 15:15:24.074596 9458 raft_consensus.cc:2454] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b07a65ba194142d78bacf332454f37f0 in term 0.
I20260923 15:15:24.074671 9258 leader_election.cc:302] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b01f4fae77b4b4da1e6b0941605e68a, b07a65ba194142d78bacf332454f37f0; no voters:
I20260923 15:15:24.074782 9529 raft_consensus.cc:2781] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:15:24.074858 9529 raft_consensus.cc:482] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:15:24.074895 9529 raft_consensus.cc:3037] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:24.075591 9529 raft_consensus.cc:504] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.075814 9529 leader_election.cc:288] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [CANDIDATE]: Term 1 election: Requested vote from peers 475b1ef5c7254855ad78fe2c6ad9b81d (127.21.204.67:35505), 3b01f4fae77b4b4da1e6b0941605e68a (127.21.204.65:41713)
I20260923 15:15:24.075870 9458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a91cb3770b64b5491080b4e95bc1656" candidate_uuid: "b07a65ba194142d78bacf332454f37f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d"
I20260923 15:15:24.075914 9190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a91cb3770b64b5491080b4e95bc1656" candidate_uuid: "b07a65ba194142d78bacf332454f37f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b01f4fae77b4b4da1e6b0941605e68a"
I20260923 15:15:24.075973 9458 raft_consensus.cc:3037] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:24.075981 9190 raft_consensus.cc:3037] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:24.076860 9458 raft_consensus.cc:2454] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b07a65ba194142d78bacf332454f37f0 in term 1.
I20260923 15:15:24.076860 9190 raft_consensus.cc:2454] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b07a65ba194142d78bacf332454f37f0 in term 1.
I20260923 15:15:24.077031 9257 leader_election.cc:302] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475b1ef5c7254855ad78fe2c6ad9b81d, b07a65ba194142d78bacf332454f37f0; no voters:
I20260923 15:15:24.077145 9529 raft_consensus.cc:2781] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:24.077394 9529 raft_consensus.cc:686] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 1 LEADER]: Becoming Leader. State: Replica: b07a65ba194142d78bacf332454f37f0, State: Running, Role: LEADER
I20260923 15:15:24.077502 9529 consensus_queue.cc:237] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } }
I20260923 15:15:24.078363 9045 catalog_manager.cc:5495] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b07a65ba194142d78bacf332454f37f0 (127.21.204.66). New cstate: current_term: 1 leader_uuid: "b07a65ba194142d78bacf332454f37f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b07a65ba194142d78bacf332454f37f0" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 44021 } health_report { overall_health: HEALTHY } } }
I20260923 15:15:24.106887 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.104435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.21.204.66:44021)
0923 15:15:24.104504 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.104506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.104516 (+ 10us) server_negotiation.cc:407] Connection header received
0923 15:15:24.104564 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.104565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.104583 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.104656 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:24.104803 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.105277 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.106086 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.106280 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.106355 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.106530 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:24.106631 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:24.106727 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.106744 (+ 17us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.106765 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 15:15:24.108695 9458 raft_consensus.cc:1261] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 1 FOLLOWER]: Refusing update from remote peer b07a65ba194142d78bacf332454f37f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:24.108698 9190 raft_consensus.cc:1261] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 1 FOLLOWER]: Refusing update from remote peer b07a65ba194142d78bacf332454f37f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:24.109073 9531 consensus_queue.cc:1035] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "475b1ef5c7254855ad78fe2c6ad9b81d" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 35505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:24.109195 9529 consensus_queue.cc:1035] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b01f4fae77b4b4da1e6b0941605e68a" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 41713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:24.112192 9534 mvcc.cc:205] Tried to move back new op lower bound from 7332563042746458112 to 7332563042621947904. Current Snapshot: MvccSnapshot[applied={T|T < 7332563042746458112 or (T in {7332563042746458112})}]
I20260923 15:15:24.112460 9536 mvcc.cc:205] Tried to move back new op lower bound from 7332563042746458112 to 7332563042621947904. Current Snapshot: MvccSnapshot[applied={T|T < 7332563042746458112}]
I20260923 15:15:24.113723 9045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:24.113826 9045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:24.114588 9045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34033960497a461b9a239d4ddd9a3d9e: Sending DeleteTablet for 3 replicas of tablet 6a91cb3770b64b5491080b4e95bc1656
I20260923 15:15:24.114928 9438 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a91cb3770b64b5491080b4e95bc1656 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:24 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:54278
I20260923 15:15:24.114977 9170 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a91cb3770b64b5491080b4e95bc1656 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:24 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:37710
I20260923 15:15:24.115008 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a91cb3770b64b5491080b4e95bc1656 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:24 UTC) from {username='kudu', principal='kudu/127.21.204.126@KRBTEST.COM'} at 127.0.0.1:50696
I20260923 15:15:24.115200 9545 tablet_replica.cc:330] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: stopping tablet replica
I20260923 15:15:24.115206 9544 tablet_replica.cc:330] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a: stopping tablet replica
I20260923 15:15:24.115293 9545 raft_consensus.cc:2229] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:15:24.115298 9544 raft_consensus.cc:2229] T 6a91cb3770b64b5491080b4e95bc1656 P 3b01f4fae77b4b4da1e6b0941605e68a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:24.115335 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9107
I20260923 15:15:24.115999 9545 raft_consensus.cc:2258] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:24.116461 9545 ts_tablet_manager.cc:1905] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:24.117026 9543 tablet_replica.cc:330] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: stopping tablet replica
I20260923 15:15:24.117170 9543 raft_consensus.cc:2229] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:24.117322 9543 raft_consensus.cc:2258] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:24.117777 9543 ts_tablet_manager.cc:1905] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:24.118305 9545 ts_tablet_manager.cc:1918] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:24.118386 9545 log.cc:1196] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/6a91cb3770b64b5491080b4e95bc1656
I20260923 15:15:24.118676 9545 ts_tablet_manager.cc:1939] T 6a91cb3770b64b5491080b4e95bc1656 P b07a65ba194142d78bacf332454f37f0: Deleting consensus metadata
I20260923 15:15:24.119141 9030 catalog_manager.cc:4841] TS b07a65ba194142d78bacf332454f37f0 (127.21.204.66:44021): tablet 6a91cb3770b64b5491080b4e95bc1656 (table test-table [id=9f538d076dd144d1b8a7fa03e64394ad]) successfully deleted
I20260923 15:15:24.119812 9543 ts_tablet_manager.cc:1918] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:24.119887 9543 log.cc:1196] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/6a91cb3770b64b5491080b4e95bc1656
I20260923 15:15:24.120134 9543 ts_tablet_manager.cc:1939] T 6a91cb3770b64b5491080b4e95bc1656 P 475b1ef5c7254855ad78fe2c6ad9b81d: Deleting consensus metadata
I20260923 15:15:24.120756 9029 catalog_manager.cc:4841] TS 475b1ef5c7254855ad78fe2c6ad9b81d (127.21.204.67:35505): tablet 6a91cb3770b64b5491080b4e95bc1656 (table test-table [id=9f538d076dd144d1b8a7fa03e64394ad]) successfully deleted
W20260923 15:15:24.122387 9030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:41713 (error 108)
I20260923 15:15:24.122704 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9241
I20260923 15:15:24.124083 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.123981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:41713 (local address 127.0.0.1:37712)
0923 15:15:24.124020 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.124054 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:41713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260923 15:15:24.124204 9030 catalog_manager.cc:4584] TS 3b01f4fae77b4b4da1e6b0941605e68a (127.21.204.65:41713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a91cb3770b64b5491080b4e95bc1656: Network error: Client connection negotiation failed: client connection to 127.21.204.65:41713: connect: Connection refused (error 111)
I20260923 15:15:24.128793 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9375
I20260923 15:15:24.133366 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9013
2026-09-23T15:15:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1957 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T15:15:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:24Z Disabled control of system clock
I20260923 15:15:24.161260 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43739
--webserver_interface=127.21.204.126
--webserver_port=0
--builtin_ntp_servers=127.21.204.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:24.241979 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:24.242182 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:24.242203 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:24.243350 9550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:24.243389 9550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:24.243402 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:24.243414 9550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:24.243459 9550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:24.244851 9550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:43739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:24.245096 9550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:24.245376 9550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:24.250029 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:24.250025 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:24.250109 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:24.250186 9550 server_base.cc:1034] running on GCE node
I20260923 15:15:24.250487 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:24.250767 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:24.251946 9550 hybrid_clock.cc:648] HybridClock initialized: now 1790176524251912 us; error 48 us; skew 500 ppm
I20260923 15:15:24.253140 9550 webserver.cc:460] Webserver started at http://127.21.204.126:34359/ using document root <none> and password file <none>
I20260923 15:15:24.253381 9550 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:24.253451 9550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:24.253604 9550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:24.254678 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "fe51cc22fe67497fa63f39c7c4ff28ba"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "111d4a536990ae5c990a74e467936f3e"
server_key_iv: "b6c373a5a51cd3461ae0631d05439d32"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.255031 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "fe51cc22fe67497fa63f39c7c4ff28ba"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "111d4a536990ae5c990a74e467936f3e"
server_key_iv: "b6c373a5a51cd3461ae0631d05439d32"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.256309 9550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 15:15:24.257109 9564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.257326 9550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:24.257427 9550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "fe51cc22fe67497fa63f39c7c4ff28ba"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "111d4a536990ae5c990a74e467936f3e"
server_key_iv: "b6c373a5a51cd3461ae0631d05439d32"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.257505 9550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:24.273743 9550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:24.274080 9550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:24.274235 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:24.277904 9550 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:43739
I20260923 15:15:24.277961 9616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:43739 every 8 connection(s)
I20260923 15:15:24.278295 9550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:24.279269 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:24.281725 9617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba: Bootstrap starting.
I20260923 15:15:24.282384 9617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:24.282707 9617 log.cc:824] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:24.283440 9617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba: No bootstrap required, opened a new log
I20260923 15:15:24.285161 9617 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43739 } }
I20260923 15:15:24.285404 9617 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:24.285406 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9550
I20260923 15:15:24.285454 9617 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe51cc22fe67497fa63f39c7c4ff28ba, State: Initialized, Role: FOLLOWER
I20260923 15:15:24.285539 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:24.285631 9617 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43739 } }
I20260923 15:15:24.285727 9617 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:24.285770 9617 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:24.285822 9617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:24.285828 22321 external_mini_cluster.cc:1428] Setting key 3b37607943ba8476b3205ece4db94514
I20260923 15:15:24.286597 9617 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43739 } }
I20260923 15:15:24.286743 9617 leader_election.cc:302] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe51cc22fe67497fa63f39c7c4ff28ba; no voters:
I20260923 15:15:24.286949 9617 leader_election.cc:288] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:24.287051 9622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:24.287235 9617 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:24.287257 9622 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [term 1 LEADER]: Becoming Leader. State: Replica: fe51cc22fe67497fa63f39c7c4ff28ba, State: Running, Role: LEADER
I20260923 15:15:24.287454 9622 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43739 } }
I20260923 15:15:24.287845 9624 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43739 } } }
I20260923 15:15:24.287967 9624 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:24.287837 9622 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe51cc22fe67497fa63f39c7c4ff28ba. Latest consensus state: current_term: 1 leader_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe51cc22fe67497fa63f39c7c4ff28ba" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 43739 } } }
I20260923 15:15:24.288166 9622 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:24.288239 9631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:24.288749 9631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:24.290416 9631 catalog_manager.cc:1347] Generated new cluster ID: 26f3ad4e261341eba7b80537bd36f91b
I20260923 15:15:24.290467 9631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:24.292109 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.286382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.21.204.126:43739)
0923 15:15:24.286609 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.286616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.286641 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:15:24.286759 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.286767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.286844 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.287026 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:24.287207 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.288890 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.289850 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.290551 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.290691 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.291374 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:24.291378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:24.291379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:24.291595 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:24.291639 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:24.291654 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:24.291728 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.291807 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.291890 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260923 15:15:24.299660 9631 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:24.300154 9631 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:24.305734 9631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba: Generated new TSK 0
I20260923 15:15:24.305928 9631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:24.313058 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.65:0
--local_ip_for_outbound_sockets=127.21.204.65
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43739
--builtin_ntp_servers=127.21.204.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:24.394461 9641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:24.394650 9641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:24.394672 9641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:24.395810 9641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:24.395853 9641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:24.395879 9641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.65
I20260923 15:15:24.397230 9641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.204.65
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.65
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:24.397465 9641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:24.397796 9641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:24.402665 9649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:24.402714 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:24.403030 9647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:24.403581 9641 server_base.cc:1034] running on GCE node
I20260923 15:15:24.403826 9641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:24.404088 9641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:24.405243 9641 hybrid_clock.cc:648] HybridClock initialized: now 1790176524405225 us; error 33 us; skew 500 ppm
I20260923 15:15:24.406576 9641 webserver.cc:460] Webserver started at http://127.21.204.65:38139/ using document root <none> and password file <none>
I20260923 15:15:24.406829 9641 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:24.406877 9641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:24.407016 9641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:24.408039 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data/instance:
uuid: "a2335a5eb543429abad29a44c0002c50"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "2a729062103d4056beab886b47ad95fc"
server_key_iv: "74d71067e2eaf220c545112902f0855d"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.408396 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance:
uuid: "a2335a5eb543429abad29a44c0002c50"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "2a729062103d4056beab886b47ad95fc"
server_key_iv: "74d71067e2eaf220c545112902f0855d"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.409698 9641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:24.410545 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.410717 9641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:24.410820 9641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/wal
uuid: "a2335a5eb543429abad29a44c0002c50"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "2a729062103d4056beab886b47ad95fc"
server_key_iv: "74d71067e2eaf220c545112902f0855d"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.410903 9641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:24.435839 9641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:24.436148 9641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:24.436301 9641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:24.436532 9641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:24.436877 9641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:24.436932 9641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.436965 9641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:24.436993 9641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.442677 9641 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.65:33443
I20260923 15:15:24.442787 9768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.65:33443 every 8 connection(s)
I20260923 15:15:24.443086 9641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/data/info.pb
I20260923 15:15:24.448266 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9641
I20260923 15:15:24.448402 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-0/wal/instance
I20260923 15:15:24.448611 22321 external_mini_cluster.cc:1428] Setting key 0058ba483a176a7c9481a2416d87bfd6
I20260923 15:15:24.448789 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.443860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:60241 (local address 127.21.204.126:43739)
0923 15:15:24.443926 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.443928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.444150 (+ 222us) server_negotiation.cc:407] Connection header received
0923 15:15:24.444678 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.444679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.444695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.444780 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:24.445537 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.446137 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.447191 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.447410 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.447476 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.448476 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:24.448478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:24.448479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:24.448484 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:24.448500 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:24.448506 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:24.448572 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.448672 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.448710 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 15:15:24.449033 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.443837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43739 (local address 127.21.204.65:60241)
0923 15:15:24.444026 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.444058 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.444554 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.444901 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.444905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.444914 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:24.445432 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.445436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.446279 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.446283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.447079 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.447082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.447514 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.447515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.447631 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.448277 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:24.448278 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:24.448305 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:24.448327 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:24.448573 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:24.448581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:24.448584 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.448674 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.448778 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260923 15:15:24.449579 9769 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43739
I20260923 15:15:24.449677 9769 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:24.449681 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.66:0
--local_ip_for_outbound_sockets=127.21.204.66
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43739
--builtin_ntp_servers=127.21.204.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 15:15:24.449915 9769 heartbeater.cc:511] Master 127.21.204.126:43739 requested a full tablet report, sending...
I20260923 15:15:24.450414 9581 ts_manager.cc:194] Registered new tserver with Master: a2335a5eb543429abad29a44c0002c50 (127.21.204.65:33443)
I20260923 15:15:24.451032 9581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.65:60241
W20260923 15:15:24.531260 9772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:24.531451 9772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:24.531472 9772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:24.532574 9772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:24.532612 9772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:24.532637 9772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.66
I20260923 15:15:24.534024 9772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.204.66
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.66
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:24.534246 9772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:24.534526 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:24.539463 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:24.539503 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:24.539490 9772 server_base.cc:1034] running on GCE node
W20260923 15:15:24.539459 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:24.539902 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:24.540199 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:24.541374 9772 hybrid_clock.cc:648] HybridClock initialized: now 1790176524541360 us; error 30 us; skew 500 ppm
I20260923 15:15:24.542685 9772 webserver.cc:460] Webserver started at http://127.21.204.66:41699/ using document root <none> and password file <none>
I20260923 15:15:24.542927 9772 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:24.542996 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:24.543131 9772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:24.544147 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data/instance:
uuid: "cb07780c8139468bb829aa9eb1a19f2f"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "8bfc02c192063c037a326fce01a9cf49"
server_key_iv: "305a4a391c826043291def6d136b2dcc"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.544510 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance:
uuid: "cb07780c8139468bb829aa9eb1a19f2f"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "8bfc02c192063c037a326fce01a9cf49"
server_key_iv: "305a4a391c826043291def6d136b2dcc"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.545804 9772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:24.546540 9786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.546697 9772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:15:24.546794 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal
uuid: "cb07780c8139468bb829aa9eb1a19f2f"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "8bfc02c192063c037a326fce01a9cf49"
server_key_iv: "305a4a391c826043291def6d136b2dcc"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.546880 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:24.558965 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:24.559233 9772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:24.559381 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:24.559608 9772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:24.559926 9772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:24.559958 9772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.559979 9772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:24.559993 9772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.565864 9772 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.66:35497
I20260923 15:15:24.565930 9899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.66:35497 every 8 connection(s)
I20260923 15:15:24.566270 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/data/info.pb
I20260923 15:15:24.571054 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.566978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.66:56113 (local address 127.21.204.126:43739)
0923 15:15:24.567056 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.567059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.567258 (+ 199us) server_negotiation.cc:407] Connection header received
0923 15:15:24.567709 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.567710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.567720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.567772 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:24.568383 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.568862 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.569789 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.570006 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.570058 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.570861 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:24.570861 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:24.570862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:24.570866 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:24.570876 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:24.570879 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:24.570917 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.570976 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.570998 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 15:15:24.571282 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.566942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43739 (local address 127.21.204.66:56113)
0923 15:15:24.567147 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.567178 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.567628 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.567821 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.567824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.567832 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:24.568300 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.568304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.568947 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.568948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.569714 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.569716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.570084 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.570085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.570193 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.570740 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:24.570741 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:24.570767 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:24.570788 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:24.570919 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:24.570921 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:24.570923 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.570966 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.571038 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260923 15:15:24.571832 9900 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43739
I20260923 15:15:24.572042 9900 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:24.572242 9900 heartbeater.cc:511] Master 127.21.204.126:43739 requested a full tablet report, sending...
I20260923 15:15:24.572638 9581 ts_manager.cc:194] Registered new tserver with Master: cb07780c8139468bb829aa9eb1a19f2f (127.21.204.66:35497)
I20260923 15:15:24.573006 9581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.66:56113
I20260923 15:15:24.574522 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9772
I20260923 15:15:24.574652 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal/instance
I20260923 15:15:24.574832 22321 external_mini_cluster.cc:1428] Setting key a1d628ebb82c1629501845e42b83e563
I20260923 15:15:24.575733 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.204.67:0
--local_ip_for_outbound_sockets=127.21.204.67
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43739
--builtin_ntp_servers=127.21.204.84:37459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 15:15:24.655903 9903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:24.656095 9903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:24.656116 9903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:24.657241 9903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:24.657279 9903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:24.657305 9903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.204.67
I20260923 15:15:24.658687 9903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:37459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.204.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.204.67
--webserver_port=0
--tserver_master_addrs=127.21.204.126:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.204.67
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:24.658950 9903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:24.659228 9903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:24.664462 9908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:24.664482 9909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:24.664490 9911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:24.664727 9903 server_base.cc:1034] running on GCE node
I20260923 15:15:24.664947 9903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:24.665198 9903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:24.666388 9903 hybrid_clock.cc:648] HybridClock initialized: now 1790176524666354 us; error 43 us; skew 500 ppm
I20260923 15:15:24.667611 9903 webserver.cc:460] Webserver started at http://127.21.204.67:38101/ using document root <none> and password file <none>
I20260923 15:15:24.667860 9903 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:24.667929 9903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:24.668079 9903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:24.669090 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data/instance:
uuid: "966babf881e7490ea353b13076e2c19f"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "a5660684796739c2438e3f3185dccca0"
server_key_iv: "8f9f7f8274a20ef1d8bbad20c68aa9f2"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.669440 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance:
uuid: "966babf881e7490ea353b13076e2c19f"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "a5660684796739c2438e3f3185dccca0"
server_key_iv: "8f9f7f8274a20ef1d8bbad20c68aa9f2"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.670819 9903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:24.671542 9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.671716 9903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 15:15:24.671813 9903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal
uuid: "966babf881e7490ea353b13076e2c19f"
format_stamp: "Formatted at 2026-09-23 15:15:24 on dist-test-slave-t5lt"
server_key: "a5660684796739c2438e3f3185dccca0"
server_key_iv: "8f9f7f8274a20ef1d8bbad20c68aa9f2"
server_key_version: "encryptionkey@0"
I20260923 15:15:24.671898 9903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:24.688136 9903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:24.688411 9903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:24.688552 9903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:24.688784 9903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 15:15:24.689119 9903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 15:15:24.689184 9903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.689224 9903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 15:15:24.689252 9903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:24.695026 9903 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.67:33923
I20260923 15:15:24.695078 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.67:33923 every 8 connection(s)
I20260923 15:15:24.695393 9903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/data/info.pb
I20260923 15:15:24.700502 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 9903
I20260923 15:15:24.700591 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.696062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.67:48093 (local address 127.21.204.126:43739)
0923 15:15:24.696107 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.696109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.696350 (+ 241us) server_negotiation.cc:407] Connection header received
0923 15:15:24.696896 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.696897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.696911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.696965 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:24.697637 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.698216 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.699201 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.699401 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.699477 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.700303 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:24.700304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:24.700304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:24.700310 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:24.700325 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:24.700332 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:24.700384 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.700484 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.700514 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:15:24.700659 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal/instance
I20260923 15:15:24.700794 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.696041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:43739 (local address 127.21.204.67:48093)
0923 15:15:24.696221 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.696254 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.696789 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.697032 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.697036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.697045 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:24.697543 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.697548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.698316 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.698317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.699095 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.699097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.699497 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.699498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.699625 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.700148 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 15:15:24.700148 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:24.700176 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 15:15:24.700198 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:24.700391 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:24.700398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:24.700401 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.700486 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.700585 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260923 15:15:24.700868 22321 external_mini_cluster.cc:1428] Setting key 8f4c2cae534d13e869a4151baff6e68a
I20260923 15:15:24.701442 10031 heartbeater.cc:347] Connected to a master server at 127.21.204.126:43739
I20260923 15:15:24.701537 10031 heartbeater.cc:464] Registering TS with master...
I20260923 15:15:24.701719 10031 heartbeater.cc:511] Master 127.21.204.126:43739 requested a full tablet report, sending...
I20260923 15:15:24.702111 9580 ts_manager.cc:194] Registered new tserver with Master: 966babf881e7490ea353b13076e2c19f (127.21.204.67:33923)
I20260923 15:15:24.702445 9580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.204.67:48093
I20260923 15:15:24.712743 22321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 15:15:24.716693 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.714181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.21.204.126:43739)
0923 15:15:24.714222 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.714223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.714315 (+ 92us) server_negotiation.cc:407] Connection header received
0923 15:15:24.714373 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.714373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.714384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.714424 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:24.714546 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.714992 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.715690 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.715874 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.715928 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.716492 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:24.716493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:24.716493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:24.716497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:24.716511 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:24.716515 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:24.716561 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.716618 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.716639 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:15:24.718564 9580 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 15:15:24.720113 9580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 15:15:24.727010 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.722456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:33923 (local address 127.0.0.1:50446)
0923 15:15:24.722920 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.722965 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.723047 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.723333 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.723337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.723340 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:24.723408 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.723411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.724761 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.724762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.725703 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.725706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.726668 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.726670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.726755 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.726769 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.726855 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.726909 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":32,"thread_start_us":72,"threads_started":1}
I20260923 15:15:24.727020 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.722629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.21.204.67:33923)
0923 15:15:24.722824 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.722830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.723035 (+ 205us) server_negotiation.cc:407] Connection header received
0923 15:15:24.723155 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.723158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.723175 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.723250 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:24.723517 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.724652 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.725792 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.726577 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.726658 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.726757 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.726873 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.726913 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260923 15:15:24.727360 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.722732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.21.204.65:33443)
0923 15:15:24.722878 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.722883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.723446 (+ 563us) server_negotiation.cc:407] Connection header received
0923 15:15:24.723493 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.723496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.723512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.723583 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:24.723777 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.724466 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.725962 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.726724 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.726786 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.726870 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.727210 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.727282 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260923 15:15:24.727142 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.722471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:33443 (local address 127.0.0.1:40458)
0923 15:15:24.723164 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.723172 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.723250 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.723656 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.723657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.723659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:24.723696 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.723697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.724557 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.724558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.725819 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.725822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.726905 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.726906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.726967 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.726971 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.727025 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.727068 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":92,"threads_started":1}
I20260923 15:15:24.727558 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.722749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.21.204.66:35497)
0923 15:15:24.722916 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.722920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.723389 (+ 469us) server_negotiation.cc:407] Connection header received
0923 15:15:24.723444 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.723447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.723463 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.723541 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:24.723650 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.724698 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.726303 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.726926 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.726988 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.727069 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.727303 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.727349 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260923 15:15:24.727352 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.722475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35497 (local address 127.0.0.1:40250)
0923 15:15:24.723328 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.723336 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.723381 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.723541 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.723542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.723544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:24.723592 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.723593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.724778 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.724779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.726217 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.726219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.727109 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.727110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.727187 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.727192 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.727258 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.727294 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":27,"thread_start_us":94,"threads_started":1}
I20260923 15:15:24.728492 9703 tablet_service.cc:1467] Processing CreateTablet for tablet aa2449588d734a8fba2a56c2873206ca (DEFAULT_TABLE table=test-table [id=dda09879085c41939a5f2f98154e9b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:24.728523 9834 tablet_service.cc:1467] Processing CreateTablet for tablet aa2449588d734a8fba2a56c2873206ca (DEFAULT_TABLE table=test-table [id=dda09879085c41939a5f2f98154e9b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:24.728516 9965 tablet_service.cc:1467] Processing CreateTablet for tablet aa2449588d734a8fba2a56c2873206ca (DEFAULT_TABLE table=test-table [id=dda09879085c41939a5f2f98154e9b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 15:15:24.728864 9703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa2449588d734a8fba2a56c2873206ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:24.728861 9834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa2449588d734a8fba2a56c2873206ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:24.728861 9965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa2449588d734a8fba2a56c2873206ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:24.731670 10045 tablet_bootstrap.cc:492] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50: Bootstrap starting.
I20260923 15:15:24.732227 10046 tablet_bootstrap.cc:492] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Bootstrap starting.
I20260923 15:15:24.732470 10045 tablet_bootstrap.cc:654] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:24.732537 10047 tablet_bootstrap.cc:492] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Bootstrap starting.
I20260923 15:15:24.732898 10046 tablet_bootstrap.cc:654] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:24.732918 10045 log.cc:824] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:24.733155 10047 tablet_bootstrap.cc:654] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:24.733224 10046 log.cc:824] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:24.733493 10047 log.cc:824] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:24.733839 10045 tablet_bootstrap.cc:492] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50: No bootstrap required, opened a new log
I20260923 15:15:24.733947 10045 ts_tablet_manager.cc:1397] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:15:24.734081 10046 tablet_bootstrap.cc:492] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: No bootstrap required, opened a new log
I20260923 15:15:24.734157 10046 ts_tablet_manager.cc:1397] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 15:15:24.734411 10047 tablet_bootstrap.cc:492] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: No bootstrap required, opened a new log
I20260923 15:15:24.734477 10047 ts_tablet_manager.cc:1397] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 15:15:24.736109 10045 raft_consensus.cc:348] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.736261 10045 raft_consensus.cc:374] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:24.736303 10045 raft_consensus.cc:729] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2335a5eb543429abad29a44c0002c50, State: Initialized, Role: FOLLOWER
I20260923 15:15:24.736410 10045 consensus_queue.cc:260] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.736675 10045 ts_tablet_manager.cc:1428] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 15:15:24.736802 10046 raft_consensus.cc:348] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.736931 9769 heartbeater.cc:503] Master 127.21.204.126:43739 was elected leader, sending a full tablet report...
I20260923 15:15:24.736963 10046 raft_consensus.cc:374] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:24.737005 10046 raft_consensus.cc:729] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb07780c8139468bb829aa9eb1a19f2f, State: Initialized, Role: FOLLOWER
I20260923 15:15:24.736937 10047 raft_consensus.cc:348] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.737072 10047 raft_consensus.cc:374] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:24.737119 10047 raft_consensus.cc:729] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966babf881e7490ea353b13076e2c19f, State: Initialized, Role: FOLLOWER
I20260923 15:15:24.737154 10046 consensus_queue.cc:260] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.737241 10047 consensus_queue.cc:260] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.737412 10046 ts_tablet_manager.cc:1428] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:15:24.737473 10047 ts_tablet_manager.cc:1428] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 15:15:24.737530 9900 heartbeater.cc:503] Master 127.21.204.126:43739 was elected leader, sending a full tablet report...
I20260923 15:15:24.737627 10031 heartbeater.cc:503] Master 127.21.204.126:43739 was elected leader, sending a full tablet report...
W20260923 15:15:24.816813 9901 tablet.cc:2369] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 15:15:24.839540 10051 raft_consensus.cc:482] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 15:15:24.839704 10051 raft_consensus.cc:504] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.840090 10051 leader_election.cc:288] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb07780c8139468bb829aa9eb1a19f2f (127.21.204.66:35497), 966babf881e7490ea353b13076e2c19f (127.21.204.67:33923)
I20260923 15:15:24.843974 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.840417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:38859 (local address 127.21.204.66:35497)
0923 15:15:24.840482 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.840484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.840493 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:24.840520 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.840521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.840529 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.840626 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:24.840795 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.841541 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.842994 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.843583 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.843650 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.843658 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.843856 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.843903 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 15:15:24.843974 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.840406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.204.65:35957 (local address 127.21.204.67:33923)
0923 15:15:24.840523 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.840526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.840616 (+ 90us) server_negotiation.cc:407] Connection header received
0923 15:15:24.840692 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.840693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.840708 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.840793 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 15:15:24.840963 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.841653 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.842994 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.843550 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.843642 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.843651 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.843856 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.843903 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 15:15:24.844017 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.840321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.66:35497 (local address 127.21.204.65:38859)
0923 15:15:24.840392 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.840405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.840451 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.840640 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.840642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.840644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:24.840719 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.840721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.841625 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.841626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.842868 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.842872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.843666 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.843668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.843758 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.843765 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.843827 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.843883 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":11}
I20260923 15:15:24.844022 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.840329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.67:33923 (local address 127.21.204.65:35957)
0923 15:15:24.840549 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.840557 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:24.840627 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:24.840804 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:24.840805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:24.840806 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 15:15:24.840878 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.840880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.841742 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.841743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.842872 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:24.842873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.843667 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.843668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:24.843758 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.843765 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 15:15:24.843827 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 15:15:24.843881 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260923 15:15:24.844485 9985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2449588d734a8fba2a56c2873206ca" candidate_uuid: "a2335a5eb543429abad29a44c0002c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "966babf881e7490ea353b13076e2c19f" is_pre_election: true
I20260923 15:15:24.844483 9854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2449588d734a8fba2a56c2873206ca" candidate_uuid: "a2335a5eb543429abad29a44c0002c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb07780c8139468bb829aa9eb1a19f2f" is_pre_election: true
I20260923 15:15:24.844636 9985 raft_consensus.cc:2454] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2335a5eb543429abad29a44c0002c50 in term 0.
I20260923 15:15:24.844643 9854 raft_consensus.cc:2454] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2335a5eb543429abad29a44c0002c50 in term 0.
I20260923 15:15:24.844856 9657 leader_election.cc:302] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 966babf881e7490ea353b13076e2c19f, a2335a5eb543429abad29a44c0002c50; no voters:
I20260923 15:15:24.844996 10051 raft_consensus.cc:2781] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 15:15:24.845055 10051 raft_consensus.cc:482] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 15:15:24.845080 10051 raft_consensus.cc:3037] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:24.845832 10051 raft_consensus.cc:504] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.846011 10051 leader_election.cc:288] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [CANDIDATE]: Term 1 election: Requested vote from peers cb07780c8139468bb829aa9eb1a19f2f (127.21.204.66:35497), 966babf881e7490ea353b13076e2c19f (127.21.204.67:33923)
I20260923 15:15:24.846164 9985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2449588d734a8fba2a56c2873206ca" candidate_uuid: "a2335a5eb543429abad29a44c0002c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "966babf881e7490ea353b13076e2c19f"
I20260923 15:15:24.846210 9854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa2449588d734a8fba2a56c2873206ca" candidate_uuid: "a2335a5eb543429abad29a44c0002c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb07780c8139468bb829aa9eb1a19f2f"
I20260923 15:15:24.846254 9985 raft_consensus.cc:3037] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:24.846326 9854 raft_consensus.cc:3037] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:24.847075 9985 raft_consensus.cc:2454] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2335a5eb543429abad29a44c0002c50 in term 1.
I20260923 15:15:24.847172 9854 raft_consensus.cc:2454] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2335a5eb543429abad29a44c0002c50 in term 1.
I20260923 15:15:24.847301 9657 leader_election.cc:302] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 966babf881e7490ea353b13076e2c19f, a2335a5eb543429abad29a44c0002c50; no voters:
I20260923 15:15:24.847385 10051 raft_consensus.cc:2781] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:24.847514 10051 raft_consensus.cc:686] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 1 LEADER]: Becoming Leader. State: Replica: a2335a5eb543429abad29a44c0002c50, State: Running, Role: LEADER
I20260923 15:15:24.847641 10051 consensus_queue.cc:237] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } }
I20260923 15:15:24.848522 9579 catalog_manager.cc:5495] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2335a5eb543429abad29a44c0002c50 (127.21.204.65). New cstate: current_term: 1 leader_uuid: "a2335a5eb543429abad29a44c0002c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2335a5eb543429abad29a44c0002c50" member_type: VOTER last_known_addr { host: "127.21.204.65" port: 33443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 } health_report { overall_health: UNKNOWN } } }
I20260923 15:15:24.874248 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.871974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.21.204.65:33443)
0923 15:15:24.872030 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:24.872032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:24.872041 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:24.872081 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:24.872082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:24.872099 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:24.872157 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:24.872275 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.872723 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.873424 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:24.873607 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:24.873661 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:24.873811 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:24.873997 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:24.874072 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:24.874108 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 15:15:24.874126 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 15:15:24.876207 9985 raft_consensus.cc:1261] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 1 FOLLOWER]: Refusing update from remote peer a2335a5eb543429abad29a44c0002c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:24.876230 9854 raft_consensus.cc:1261] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 1 FOLLOWER]: Refusing update from remote peer a2335a5eb543429abad29a44c0002c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:24.876581 10051 consensus_queue.cc:1035] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb07780c8139468bb829aa9eb1a19f2f" member_type: VOTER last_known_addr { host: "127.21.204.66" port: 35497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:24.876721 10055 consensus_queue.cc:1035] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "966babf881e7490ea353b13076e2c19f" member_type: VOTER last_known_addr { host: "127.21.204.67" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:24.879212 10058 mvcc.cc:205] Tried to move back new op lower bound from 7332563045889814528 to 7332563045776609280. Current Snapshot: MvccSnapshot[applied={T|T < 7332563045889814528}]
I20260923 15:15:24.879292 10060 mvcc.cc:205] Tried to move back new op lower bound from 7332563045889814528 to 7332563045776609280. Current Snapshot: MvccSnapshot[applied={T|T < 7332563045889814528}]
I20260923 15:15:24.879835 10059 mvcc.cc:205] Tried to move back new op lower bound from 7332563045889814528 to 7332563045776609280. Current Snapshot: MvccSnapshot[applied={T|T < 7332563045889814528}]
I20260923 15:15:24.881302 9580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:24.881392 9580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 15:15:24.882225 9580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe51cc22fe67497fa63f39c7c4ff28ba: Sending DeleteTablet for 3 replicas of tablet aa2449588d734a8fba2a56c2873206ca
I20260923 15:15:24.882579 9703 tablet_service.cc:1514] Processing DeleteTablet for tablet aa2449588d734a8fba2a56c2873206ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:24 UTC) from {username='slave'} at 127.0.0.1:40458
I20260923 15:15:24.882604 9834 tablet_service.cc:1514] Processing DeleteTablet for tablet aa2449588d734a8fba2a56c2873206ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:24 UTC) from {username='slave'} at 127.0.0.1:40250
I20260923 15:15:24.882589 9965 tablet_service.cc:1514] Processing DeleteTablet for tablet aa2449588d734a8fba2a56c2873206ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 15:15:24 UTC) from {username='slave'} at 127.0.0.1:50446
I20260923 15:15:24.882827 10068 tablet_replica.cc:330] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: stopping tablet replica
I20260923 15:15:24.882906 10070 tablet_replica.cc:330] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50: stopping tablet replica
I20260923 15:15:24.883018 10070 raft_consensus.cc:2229] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 1 LEADER]: Raft consensus shutting down.
I20260923 15:15:24.883041 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9641
I20260923 15:15:24.883023 10069 tablet_replica.cc:330] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: stopping tablet replica
I20260923 15:15:24.883141 10070 raft_consensus.cc:2258] T aa2449588d734a8fba2a56c2873206ca P a2335a5eb543429abad29a44c0002c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:24.883263 10069 raft_consensus.cc:2229] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:24.883380 10068 raft_consensus.cc:2229] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 15:15:24.883508 10069 raft_consensus.cc:2258] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:24.883630 10068 raft_consensus.cc:2258] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 15:15:24.884176 10069 ts_tablet_manager.cc:1905] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:24.884323 10068 ts_tablet_manager.cc:1905] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 15:15:24.885828 10069 ts_tablet_manager.cc:1918] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:24.885939 10069 log.cc:1196] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-2/wal/wals/aa2449588d734a8fba2a56c2873206ca
I20260923 15:15:24.886071 10068 ts_tablet_manager.cc:1918] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 15:15:24.886193 10068 log.cc:1196] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Deleting WAL directory at /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790176278407460-22321-0/minicluster-data/ts-1/wal/wals/aa2449588d734a8fba2a56c2873206ca
I20260923 15:15:24.886202 10069 ts_tablet_manager.cc:1939] T aa2449588d734a8fba2a56c2873206ca P 966babf881e7490ea353b13076e2c19f: Deleting consensus metadata
I20260923 15:15:24.886487 10068 ts_tablet_manager.cc:1939] T aa2449588d734a8fba2a56c2873206ca P cb07780c8139468bb829aa9eb1a19f2f: Deleting consensus metadata
I20260923 15:15:24.886633 9566 catalog_manager.cc:4841] TS 966babf881e7490ea353b13076e2c19f (127.21.204.67:33923): tablet aa2449588d734a8fba2a56c2873206ca (table test-table [id=dda09879085c41939a5f2f98154e9b59]) successfully deleted
I20260923 15:15:24.886957 9568 catalog_manager.cc:4841] TS cb07780c8139468bb829aa9eb1a19f2f (127.21.204.66:35497): tablet aa2449588d734a8fba2a56c2873206ca (table test-table [id=dda09879085c41939a5f2f98154e9b59]) successfully deleted
W20260923 15:15:24.889714 9567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.204.65:33443 (error 108)
I20260923 15:15:24.889926 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9772
I20260923 15:15:24.890596 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:24.890067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.65:33443 (local address 127.0.0.1:40476)
0923 15:15:24.890514 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:24.890557 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.65:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425}
W20260923 15:15:24.890743 9567 catalog_manager.cc:4584] TS a2335a5eb543429abad29a44c0002c50 (127.21.204.65:33443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa2449588d734a8fba2a56c2873206ca: Network error: Client connection negotiation failed: client connection to 127.21.204.65:33443: connect: Connection refused (error 111)
I20260923 15:15:24.895298 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9903
I20260923 15:15:24.900781 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 9550
2026-09-23T15:15:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (768 ms)
[----------] 6 tests from EncryptionPolicyTest (8938 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:24 dist-test-slave-t5lt krb5kdc[10074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:24 dist-test-slave-t5lt krb5kdc[10074](info): set up 2 sockets
Sep 23 15:15:24 dist-test-slave-t5lt krb5kdc[10074](info): commencing operation
krb5kdc: starting...
W20260923 15:15:25.951198 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:26 dist-test-slave-t5lt krb5kdc[10074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:26.070329 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:26.163743 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:26.163967 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:26.164014 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:26.165148 10087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:26.165213 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:26.165238 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:26.165264 10087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:26.165297 10087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:26.166772 10087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:26.167042 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:26.167260 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:26.172026 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:26.172057 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:26.172192 10087 server_base.cc:1034] running on GCE node
W20260923 15:15:26.172039 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:26.172482 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:26.172781 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:26.173957 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790176526173928 us; error 33 us; skew 500 ppm
Sep 23 15:15:26 dist-test-slave-t5lt krb5kdc[10074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:26.176345 10087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:26.176757 10087 webserver.cc:460] Webserver started at http://127.0.0.1:37989/ using document root <none> and password file <none>
I20260923 15:15:26.176972 10087 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:26.177039 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:26.177188 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:26.178293 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "f6f0e562d2e040088900cb0ea5b127d5"
format_stamp: "Formatted at 2026-09-23 15:15:26 on dist-test-slave-t5lt"
server_key: "3a5b052e0399f8df7e14ade3edaf04dc"
server_key_iv: "e4ec6ede8e47ea0b419235ca1d04df17"
server_key_version: "encryptionkey@0"
I20260923 15:15:26.178639 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "f6f0e562d2e040088900cb0ea5b127d5"
format_stamp: "Formatted at 2026-09-23 15:15:26 on dist-test-slave-t5lt"
server_key: "3a5b052e0399f8df7e14ade3edaf04dc"
server_key_iv: "e4ec6ede8e47ea0b419235ca1d04df17"
server_key_version: "encryptionkey@0"
I20260923 15:15:26.179997 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:15:26.180785 10102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:26.180963 10087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:15:26.181082 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "f6f0e562d2e040088900cb0ea5b127d5"
format_stamp: "Formatted at 2026-09-23 15:15:26 on dist-test-slave-t5lt"
server_key: "3a5b052e0399f8df7e14ade3edaf04dc"
server_key_iv: "e4ec6ede8e47ea0b419235ca1d04df17"
server_key_version: "encryptionkey@0"
I20260923 15:15:26.181173 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:26.196951 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:26.199430 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:26.199632 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:26.204974 10087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38073
I20260923 15:15:26.205010 10154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38073 every 8 connection(s)
I20260923 15:15:26.205365 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:26.206290 10155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:26.208559 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5: Bootstrap starting.
I20260923 15:15:26.209144 10155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:26.209445 10155 log.cc:824] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:26.210201 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5: No bootstrap required, opened a new log
I20260923 15:15:26.211772 10155 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f0e562d2e040088900cb0ea5b127d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38073 } }
I20260923 15:15:26.211907 10155 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:26.211930 10155 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f0e562d2e040088900cb0ea5b127d5, State: Initialized, Role: FOLLOWER
I20260923 15:15:26.212105 10155 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f0e562d2e040088900cb0ea5b127d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38073 } }
I20260923 15:15:26.212167 10155 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:26.212188 10155 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:26.212236 10155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:26.212730 10155 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f0e562d2e040088900cb0ea5b127d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38073 } }
I20260923 15:15:26.212837 10155 leader_election.cc:302] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f0e562d2e040088900cb0ea5b127d5; no voters:
I20260923 15:15:26.213018 10155 leader_election.cc:288] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:26.213099 10158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:26.213296 10155 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:26.213294 10158 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [term 1 LEADER]: Becoming Leader. State: Replica: f6f0e562d2e040088900cb0ea5b127d5, State: Running, Role: LEADER
I20260923 15:15:26.213434 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f0e562d2e040088900cb0ea5b127d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38073 } }
I20260923 15:15:26.213786 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10087
I20260923 15:15:26.213814 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f0e562d2e040088900cb0ea5b127d5. Latest consensus state: current_term: 1 leader_uuid: "f6f0e562d2e040088900cb0ea5b127d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f0e562d2e040088900cb0ea5b127d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38073 } } }
I20260923 15:15:26.213914 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:26.213969 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:26.214061 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f0e562d2e040088900cb0ea5b127d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f0e562d2e040088900cb0ea5b127d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38073 } } }
I20260923 15:15:26.214130 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:26.214184 22321 external_mini_cluster.cc:1428] Setting key 10712f0429b3d2f5543e87c9c7852ef6
I20260923 15:15:26.214267 10162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:26.215639 10162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:26.217104 10162 catalog_manager.cc:1347] Generated new cluster ID: b1963caeacf54b9ab94e3a341ace5665
I20260923 15:15:26.217830 10162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:26 dist-test-slave-t5lt krb5kdc[10074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 15:15:26.225335 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:26.214733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33089 (local address 127.0.0.1:38073)
0923 15:15:26.215303 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:26.215314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:26.215348 (+ 34us) server_negotiation.cc:407] Connection header received
0923 15:15:26.215470 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:26.215476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:26.215518 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:26.215668 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:26.216351 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:26.217515 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:26.218420 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:26.218840 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:26.218954 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:26.221292 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:26.221296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:26.221297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:26.221311 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:26.223607 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:26.224094 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:26.224097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:26.224098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:26.224250 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:26.224503 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:26.224504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:26.224505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:26.224742 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:26.224928 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:26.225014 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 15:15:26.225130 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":133,"threads_started":1}
I20260923 15:15:26.226300 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:26.232692 10162 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:26.233348 10162 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:26.234073 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:26.227611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36817 (local address 127.0.0.1:38073)
0923 15:15:26.227649 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:26.227650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:26.227772 (+ 122us) server_negotiation.cc:407] Connection header received
0923 15:15:26.227835 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:26.227836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:26.227846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:26.227888 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:26.228386 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:26.228906 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:26.229762 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:26.230055 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:26.230131 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:26.231265 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:26.231266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:26.231266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:26.231273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:26.232833 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:26.233198 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:26.233200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:26.233200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:26.233254 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:26.233500 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:26.233500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:26.233501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:26.233609 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:26.233717 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:26.233844 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 15:15:26.233938 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:15:26.244252 10162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f0e562d2e040088900cb0ea5b127d5: Generated new TSK 0
I20260923 15:15:26.244447 10162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:26.246613 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10087
W20260923 15:15:26.251719 10072 connection.cc:535] client connection to 127.0.0.1:38073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38073): Connection reset by peer (error 104)
2026-09-23T15:15:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:26 dist-test-slave-t5lt krb5kdc[10186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:26 dist-test-slave-t5lt krb5kdc[10186](info): set up 2 sockets
Sep 23 15:15:26 dist-test-slave-t5lt krb5kdc[10186](info): commencing operation
krb5kdc: starting...
W20260923 15:15:27.287302 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:27 dist-test-slave-t5lt krb5kdc[10186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:27.404819 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:27.484999 10199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:27.485232 10199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:27.485273 10199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:27.486500 10199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:27.486564 10199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:27.486590 10199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:27.486622 10199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:27.486654 10199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:27.488121 10199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:27.488397 10199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:27.488615 10199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:27.493309 10204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:27.493315 10205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:27.493335 10207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:27.493804 10199 server_base.cc:1034] running on GCE node
I20260923 15:15:27.494097 10199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:27.494401 10199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:27.495584 10199 hybrid_clock.cc:648] HybridClock initialized: now 1790176527495569 us; error 29 us; skew 500 ppm
Sep 23 15:15:27 dist-test-slave-t5lt krb5kdc[10186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:27.498162 10199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:27.498667 10199 webserver.cc:460] Webserver started at http://127.0.0.1:38845/ using document root <none> and password file <none>
I20260923 15:15:27.498905 10199 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:27.498976 10199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:27.499110 10199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:27.500135 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "f4a1cce5e74d4930a49ec9225491b77f"
format_stamp: "Formatted at 2026-09-23 15:15:27 on dist-test-slave-t5lt"
server_key: "07c3e9582e9153fe05715588ba2fb9fc"
server_key_iv: "f70dbfb102b2663179a3bf02f0930bd3"
server_key_version: "encryptionkey@0"
I20260923 15:15:27.500506 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "f4a1cce5e74d4930a49ec9225491b77f"
format_stamp: "Formatted at 2026-09-23 15:15:27 on dist-test-slave-t5lt"
server_key: "07c3e9582e9153fe05715588ba2fb9fc"
server_key_iv: "f70dbfb102b2663179a3bf02f0930bd3"
server_key_version: "encryptionkey@0"
I20260923 15:15:27.501941 10199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:27.502738 10214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:27.502936 10199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:27.503034 10199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "f4a1cce5e74d4930a49ec9225491b77f"
format_stamp: "Formatted at 2026-09-23 15:15:27 on dist-test-slave-t5lt"
server_key: "07c3e9582e9153fe05715588ba2fb9fc"
server_key_iv: "f70dbfb102b2663179a3bf02f0930bd3"
server_key_version: "encryptionkey@0"
I20260923 15:15:27.503121 10199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:27.514824 10199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:27.516505 10199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:27.516664 10199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:27.520777 10199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45249
I20260923 15:15:27.520807 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45249 every 8 connection(s)
I20260923 15:15:27.521138 10199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:27.522022 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:27.524152 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f: Bootstrap starting.
I20260923 15:15:27.524720 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:27.525008 10267 log.cc:824] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:27.525689 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f: No bootstrap required, opened a new log
I20260923 15:15:27.527500 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a1cce5e74d4930a49ec9225491b77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260923 15:15:27.527630 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:27.527652 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a1cce5e74d4930a49ec9225491b77f, State: Initialized, Role: FOLLOWER
I20260923 15:15:27.527854 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a1cce5e74d4930a49ec9225491b77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260923 15:15:27.527940 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:27.527985 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:27.528038 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:27.528545 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a1cce5e74d4930a49ec9225491b77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260923 15:15:27.528661 10267 leader_election.cc:302] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4a1cce5e74d4930a49ec9225491b77f; no voters:
I20260923 15:15:27.528836 10267 leader_election.cc:288] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:27.528913 10270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:27.529063 10270 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [term 1 LEADER]: Becoming Leader. State: Replica: f4a1cce5e74d4930a49ec9225491b77f, State: Running, Role: LEADER
I20260923 15:15:27.529084 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:27.529153 10270 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a1cce5e74d4930a49ec9225491b77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260923 15:15:27.529202 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10199
I20260923 15:15:27.529330 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:27.529498 10270 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a1cce5e74d4930a49ec9225491b77f. Latest consensus state: current_term: 1 leader_uuid: "f4a1cce5e74d4930a49ec9225491b77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a1cce5e74d4930a49ec9225491b77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } } }
I20260923 15:15:27.529541 22321 external_mini_cluster.cc:1428] Setting key 2de9c37204bb79d42f5b7fa2900593d6
I20260923 15:15:27.529562 10270 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:27.529657 10271 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4a1cce5e74d4930a49ec9225491b77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a1cce5e74d4930a49ec9225491b77f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } } }
I20260923 15:15:27.529740 10271 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:27.530646 10277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:27.531210 10277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:27.533326 10277 catalog_manager.cc:1347] Generated new cluster ID: 6d8ae68a3e3d456c9690276a8f60c8e6
I20260923 15:15:27.533418 10277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:27.535395 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:27.530024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45307 (local address 127.0.0.1:45249)
0923 15:15:27.530215 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:27.530226 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:27.530366 (+ 140us) server_negotiation.cc:407] Connection header received
0923 15:15:27.530524 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:27.530531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:27.530594 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:27.530727 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:27.530997 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:27.532424 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:27.533261 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:27.533900 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:27.533998 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:27.534713 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:27.534717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:27.534718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:27.534913 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:27.534949 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:27.534961 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:27.535036 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:27.535155 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 15:15:27.535181 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260923 15:15:27.536396 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:27.541388 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:27.537949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50153 (local address 127.0.0.1:45249)
0923 15:15:27.537989 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:27.537991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:27.538145 (+ 154us) server_negotiation.cc:407] Connection header received
0923 15:15:27.538201 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:27.538202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:27.538212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:27.538258 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:27.538393 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:27.539101 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:27.540130 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:27.540433 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:27.540488 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:27.541143 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:27.541144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:27.541144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:27.541148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:27.541160 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:27.541164 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:27.541216 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:27.541311 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 15:15:27.541322 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:15:27.555603 10277 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:27.556161 10277 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:27.560601 10277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4a1cce5e74d4930a49ec9225491b77f: Generated new TSK 0
I20260923 15:15:27.560778 10277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:27.565550 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10199
W20260923 15:15:27.570039 10184 connection.cc:535] client connection to 127.0.0.1:45249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45249): Connection reset by peer (error 104)
2026-09-23T15:15:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:27 dist-test-slave-t5lt krb5kdc[10298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:27 dist-test-slave-t5lt krb5kdc[10298](info): set up 2 sockets
Sep 23 15:15:27 dist-test-slave-t5lt krb5kdc[10298](info): commencing operation
krb5kdc: starting...
W20260923 15:15:28.595498 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:28 dist-test-slave-t5lt krb5kdc[10298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:28.713069 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:28.792315 10311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:28.792539 10311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:28.792575 10311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:28.793696 10311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:28.793764 10311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:28.793790 10311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:28.793819 10311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:28.793843 10311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:28.795293 10311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:28.795567 10311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:28.795773 10311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:28.800626 10316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:28.800640 10317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:28.800640 10319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:28.800952 10311 server_base.cc:1034] running on GCE node
I20260923 15:15:28.801187 10311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:28.801476 10311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:28.802630 10311 hybrid_clock.cc:648] HybridClock initialized: now 1790176528802616 us; error 27 us; skew 500 ppm
Sep 23 15:15:28 dist-test-slave-t5lt krb5kdc[10298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:28.805051 10311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:28.805462 10311 webserver.cc:460] Webserver started at http://127.0.0.1:39819/ using document root <none> and password file <none>
I20260923 15:15:28.805683 10311 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:28.805751 10311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:28.805950 10311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:28.807000 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "cc616fdce1e54ebbac845b9f3e18d44f"
format_stamp: "Formatted at 2026-09-23 15:15:28 on dist-test-slave-t5lt"
server_key: "99d7673e000442fb4ea3fcb889c8f1a0"
server_key_iv: "2f795b27baf2dd66919ee997871fde3b"
server_key_version: "encryptionkey@0"
I20260923 15:15:28.807358 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "cc616fdce1e54ebbac845b9f3e18d44f"
format_stamp: "Formatted at 2026-09-23 15:15:28 on dist-test-slave-t5lt"
server_key: "99d7673e000442fb4ea3fcb889c8f1a0"
server_key_iv: "2f795b27baf2dd66919ee997871fde3b"
server_key_version: "encryptionkey@0"
I20260923 15:15:28.808718 10311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:28.809530 10326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:28.809746 10311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:28.809847 10311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "cc616fdce1e54ebbac845b9f3e18d44f"
format_stamp: "Formatted at 2026-09-23 15:15:28 on dist-test-slave-t5lt"
server_key: "99d7673e000442fb4ea3fcb889c8f1a0"
server_key_iv: "2f795b27baf2dd66919ee997871fde3b"
server_key_version: "encryptionkey@0"
I20260923 15:15:28.809978 10311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:28.813154 10311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:28.815050 10311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:28.815210 10311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:28.819082 10311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36965
I20260923 15:15:28.819111 10378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36965 every 8 connection(s)
I20260923 15:15:28.819433 10311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:28.820423 10379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:28.822629 10379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f: Bootstrap starting.
I20260923 15:15:28.823189 10379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:28.823477 10379 log.cc:824] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:28.824136 10379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f: No bootstrap required, opened a new log
I20260923 15:15:28.825654 10379 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36965 } }
I20260923 15:15:28.825780 10379 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:28.825827 10379 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc616fdce1e54ebbac845b9f3e18d44f, State: Initialized, Role: FOLLOWER
I20260923 15:15:28.826028 10379 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36965 } }
I20260923 15:15:28.826097 10379 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:28.826117 10379 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:28.826185 10379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:28.826673 10379 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36965 } }
I20260923 15:15:28.826781 10379 leader_election.cc:302] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc616fdce1e54ebbac845b9f3e18d44f; no voters:
I20260923 15:15:28.826970 10379 leader_election.cc:288] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:28.827036 10382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:28.827183 10382 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [term 1 LEADER]: Becoming Leader. State: Replica: cc616fdce1e54ebbac845b9f3e18d44f, State: Running, Role: LEADER
I20260923 15:15:28.827211 10379 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:28.827381 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10311
I20260923 15:15:28.827487 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:28.827657 22321 external_mini_cluster.cc:1428] Setting key b3fd4d142a2e68d16489d692a3e2db8a
I20260923 15:15:28.827299 10382 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36965 } }
I20260923 15:15:28.828663 10384 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc616fdce1e54ebbac845b9f3e18d44f. Latest consensus state: current_term: 1 leader_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36965 } } }
I20260923 15:15:28.828761 10384 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:28.829098 10383 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc616fdce1e54ebbac845b9f3e18d44f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36965 } } }
I20260923 15:15:28.829159 10383 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f [sys.catalog]: This master's current role is: LEADER
W20260923 15:15:28.829942 10399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:15:28.830008 10399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:28.830137 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:28.828119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34989 (local address 127.0.0.1:36965)
0923 15:15:28.828466 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:28.828474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:28.828503 (+ 29us) server_negotiation.cc:407] Connection header received
0923 15:15:28.828622 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:28.828630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:28.828670 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:28.829108 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:28.829520 (+ 412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:28.829524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:28.829524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:28.829528 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:28.829572 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:28.829588 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:28.829650 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:28.829733 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 15:15:28.829902 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":80,"threads_started":1}
I20260923 15:15:28.831032 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:28.833545 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:28.832580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33983 (local address 127.0.0.1:36965)
0923 15:15:28.832618 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:28.832620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:28.832692 (+ 72us) server_negotiation.cc:407] Connection header received
0923 15:15:28.832754 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:28.832754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:28.832758 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:28.832826 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:28.833310 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:28.833311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:28.833311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:28.833314 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:28.833326 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:28.833329 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:28.833372 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:28.833430 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 15:15:28.833447 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:15:28.837977 10396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:28.838213 10396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:28.839649 10396 catalog_manager.cc:1347] Generated new cluster ID: 784c1c4f1dfd4e4f976f621c80365398
I20260923 15:15:28.839702 10396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:28.876200 10396 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:28.876422 10396 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:28.883262 10396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc616fdce1e54ebbac845b9f3e18d44f: Generated new TSK 0
I20260923 15:15:28.883468 10396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:28.893594 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10311
2026-09-23T15:15:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:28 dist-test-slave-t5lt krb5kdc[10410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:28 dist-test-slave-t5lt krb5kdc[10410](info): set up 2 sockets
Sep 23 15:15:28 dist-test-slave-t5lt krb5kdc[10410](info): commencing operation
krb5kdc: starting...
W20260923 15:15:29.939278 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:30 dist-test-slave-t5lt krb5kdc[10410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:30.057102 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:30.138504 10423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:30.138726 10423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:30.138765 10423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:30.139914 10423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:30.139982 10423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:30.140007 10423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:30.140038 10423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:30.140062 10423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:30.141543 10423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:30.141814 10423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:30.142052 10423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:30.146916 10429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:30.146962 10428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:30.146978 10431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:30.147120 10423 server_base.cc:1034] running on GCE node
I20260923 15:15:30.147414 10423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:30.147718 10423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:30.148905 10423 hybrid_clock.cc:648] HybridClock initialized: now 1790176530148868 us; error 48 us; skew 500 ppm
Sep 23 15:15:30 dist-test-slave-t5lt krb5kdc[10410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:30.151530 10423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:30.151971 10423 webserver.cc:460] Webserver started at http://127.0.0.1:38613/ using document root <none> and password file <none>
I20260923 15:15:30.152215 10423 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:30.152287 10423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:30.152436 10423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:30.153465 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "473e48f6fa514d11aa85b45e17a03ef9"
format_stamp: "Formatted at 2026-09-23 15:15:30 on dist-test-slave-t5lt"
server_key: "f711d55d2d90c40dfd8c3352a8a63d69"
server_key_iv: "13191656f9f4c65dc3d4e1ff715d8915"
server_key_version: "encryptionkey@0"
I20260923 15:15:30.153825 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "473e48f6fa514d11aa85b45e17a03ef9"
format_stamp: "Formatted at 2026-09-23 15:15:30 on dist-test-slave-t5lt"
server_key: "f711d55d2d90c40dfd8c3352a8a63d69"
server_key_iv: "13191656f9f4c65dc3d4e1ff715d8915"
server_key_version: "encryptionkey@0"
I20260923 15:15:30.155185 10423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:30.155948 10438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:30.156322 10423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:30.156469 10423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "473e48f6fa514d11aa85b45e17a03ef9"
format_stamp: "Formatted at 2026-09-23 15:15:30 on dist-test-slave-t5lt"
server_key: "f711d55d2d90c40dfd8c3352a8a63d69"
server_key_iv: "13191656f9f4c65dc3d4e1ff715d8915"
server_key_version: "encryptionkey@0"
I20260923 15:15:30.156554 10423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:30.172128 10423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:30.174064 10423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:30.174230 10423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:30.178320 10423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40291
I20260923 15:15:30.178354 10490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40291 every 8 connection(s)
I20260923 15:15:30.178709 10423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:30.179630 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:30.181784 10491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9: Bootstrap starting.
I20260923 15:15:30.181981 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10423
I20260923 15:15:30.182116 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:30.182283 22321 external_mini_cluster.cc:1428] Setting key dd3bff7707baee27d7a61978828c1743
I20260923 15:15:30.182518 10491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:30.182850 10491 log.cc:824] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:30.183660 10491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9: No bootstrap required, opened a new log
I20260923 15:15:30.186036 10491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473e48f6fa514d11aa85b45e17a03ef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260923 15:15:30.186199 10491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:30.186231 10491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473e48f6fa514d11aa85b45e17a03ef9, State: Initialized, Role: FOLLOWER
I20260923 15:15:30.186375 10491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473e48f6fa514d11aa85b45e17a03ef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260923 15:15:30.186491 10491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:30.186537 10491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:30.186587 10491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:30.187202 10491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473e48f6fa514d11aa85b45e17a03ef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260923 15:15:30.187306 10491 leader_election.cc:302] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473e48f6fa514d11aa85b45e17a03ef9; no voters:
I20260923 15:15:30.187454 10491 leader_election.cc:288] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:30.187527 10496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:30.187702 10491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:30.187711 10496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [term 1 LEADER]: Becoming Leader. State: Replica: 473e48f6fa514d11aa85b45e17a03ef9, State: Running, Role: LEADER
I20260923 15:15:30.187907 10496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473e48f6fa514d11aa85b45e17a03ef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260923 15:15:30.188470 10497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473e48f6fa514d11aa85b45e17a03ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473e48f6fa514d11aa85b45e17a03ef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } } }
I20260923 15:15:30.188490 10498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473e48f6fa514d11aa85b45e17a03ef9. Latest consensus state: current_term: 1 leader_uuid: "473e48f6fa514d11aa85b45e17a03ef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473e48f6fa514d11aa85b45e17a03ef9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } } }
I20260923 15:15:30.188745 10497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:30.188752 10498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:30.188997 10506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:30.189620 10506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 15:15:30 dist-test-slave-t5lt krb5kdc[10410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 15:15:30.193979 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:30.182851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49661 (local address 127.0.0.1:40291)
0923 15:15:30.183146 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:30.183155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:30.183241 (+ 86us) server_negotiation.cc:407] Connection header received
0923 15:15:30.183904 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:30.183912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:30.183984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:30.184131 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:30.184795 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:30.186241 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:30.187089 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:30.187700 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:30.188163 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:30.190826 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:30.190829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:30.190829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:30.190839 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:30.192391 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:30.192792 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:30.192794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:30.192794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:30.192881 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:30.193095 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:30.193096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:30.193096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:30.193306 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:30.193423 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:30.193558 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 15:15:30.193658 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20260923 15:15:30.195024 10506 catalog_manager.cc:1347] Generated new cluster ID: 5b02abc306ee4eb9965759f7167d7f71
I20260923 15:15:30.195078 10506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:30.195062 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:30.201807 10506 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:30.202548 10506 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:30.204836 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:30.197738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35161 (local address 127.0.0.1:40291)
0923 15:15:30.197793 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:30.197795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:30.198093 (+ 298us) server_negotiation.cc:407] Connection header received
0923 15:15:30.198172 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:30.198173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:30.198182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:30.198206 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:30.198796 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:30.199505 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:30.200479 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:30.200781 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:30.200816 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:30.202274 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:30.202276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:30.202276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:30.202282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:30.203784 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:30.204135 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:30.204136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:30.204136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:30.204183 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:30.204342 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:30.204342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:30.204342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:30.204442 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:30.204505 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:30.204674 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 15:15:30.204755 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 15:15:30.207261 10506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473e48f6fa514d11aa85b45e17a03ef9: Generated new TSK 0
I20260923 15:15:30.207423 10506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:30.211378 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10423
2026-09-23T15:15:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:30 dist-test-slave-t5lt krb5kdc[10522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:30 dist-test-slave-t5lt krb5kdc[10522](info): set up 2 sockets
Sep 23 15:15:30 dist-test-slave-t5lt krb5kdc[10522](info): commencing operation
krb5kdc: starting...
W20260923 15:15:31.251232 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:31 dist-test-slave-t5lt krb5kdc[10522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:31.367048 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:31.450085 10535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:31.450325 10535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:31.450363 10535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:31.451519 10535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:31.451586 10535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:31.451611 10535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:31.451643 10535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:31.451668 10535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:31.453137 10535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:31.453405 10535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:31.453634 10535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:31.458700 10540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:31.458724 10543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:31.458719 10541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:31.458761 10535 server_base.cc:1034] running on GCE node
I20260923 15:15:31.459175 10535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:31.459455 10535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:31.460625 10535 hybrid_clock.cc:648] HybridClock initialized: now 1790176531460610 us; error 31 us; skew 500 ppm
Sep 23 15:15:31 dist-test-slave-t5lt krb5kdc[10522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:31.468742 10535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:31.469238 10535 webserver.cc:460] Webserver started at http://127.0.0.1:41149/ using document root <none> and password file <none>
I20260923 15:15:31.469434 10535 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:31.469537 10535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:31.469700 10535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:31.470872 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "560a3b769e034691b690503ed7836fdf"
format_stamp: "Formatted at 2026-09-23 15:15:31 on dist-test-slave-t5lt"
server_key: "f2cdf460c5e5d26add2b3007522d2d81"
server_key_iv: "1e81bb7304542314905f8b9f5df99fcc"
server_key_version: "encryptionkey@0"
I20260923 15:15:31.471267 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "560a3b769e034691b690503ed7836fdf"
format_stamp: "Formatted at 2026-09-23 15:15:31 on dist-test-slave-t5lt"
server_key: "f2cdf460c5e5d26add2b3007522d2d81"
server_key_iv: "1e81bb7304542314905f8b9f5df99fcc"
server_key_version: "encryptionkey@0"
I20260923 15:15:31.472663 10535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:15:31.473457 10550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:31.473654 10535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:31.473747 10535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "560a3b769e034691b690503ed7836fdf"
format_stamp: "Formatted at 2026-09-23 15:15:31 on dist-test-slave-t5lt"
server_key: "f2cdf460c5e5d26add2b3007522d2d81"
server_key_iv: "1e81bb7304542314905f8b9f5df99fcc"
server_key_version: "encryptionkey@0"
I20260923 15:15:31.473795 10535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:31.489082 10535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:31.491006 10535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:31.491115 10535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:31.495126 10535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34797
I20260923 15:15:31.495167 10602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34797 every 8 connection(s)
I20260923 15:15:31.495501 10535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:31.496335 10603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:31.498730 10603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf: Bootstrap starting.
I20260923 15:15:31.499306 10603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:31.499605 10603 log.cc:824] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:31.500285 10603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf: No bootstrap required, opened a new log
I20260923 15:15:31.501726 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10535
I20260923 15:15:31.501857 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:31.502063 22321 external_mini_cluster.cc:1428] Setting key d8e7de4aefcff840f7011a2d780707ab
I20260923 15:15:31.501997 10603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a3b769e034691b690503ed7836fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260923 15:15:31.502161 10603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:31.502209 10603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560a3b769e034691b690503ed7836fdf, State: Initialized, Role: FOLLOWER
I20260923 15:15:31.502380 10603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a3b769e034691b690503ed7836fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260923 15:15:31.502486 10603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:31.502519 10603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:31.502560 10603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:31.503120 10603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a3b769e034691b690503ed7836fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260923 15:15:31.503239 10603 leader_election.cc:302] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560a3b769e034691b690503ed7836fdf; no voters:
I20260923 15:15:31.503417 10603 leader_election.cc:288] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:31.503435 10608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:31.503547 10608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [term 1 LEADER]: Becoming Leader. State: Replica: 560a3b769e034691b690503ed7836fdf, State: Running, Role: LEADER
I20260923 15:15:31.503669 10603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:31.503679 10608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a3b769e034691b690503ed7836fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260923 15:15:31.504191 10609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "560a3b769e034691b690503ed7836fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a3b769e034691b690503ed7836fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } } }
I20260923 15:15:31.504289 10609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:31.504495 10610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 560a3b769e034691b690503ed7836fdf. Latest consensus state: current_term: 1 leader_uuid: "560a3b769e034691b690503ed7836fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560a3b769e034691b690503ed7836fdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } } }
I20260923 15:15:31.504762 10610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:31.504980 10616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:31.505434 10616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:31.507026 10616 catalog_manager.cc:1347] Generated new cluster ID: bb988650df904c2a94ea00ece5b7670a
I20260923 15:15:31.507078 10616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:31.508203 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:31.502560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45087 (local address 127.0.0.1:34797)
0923 15:15:31.502746 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:31.502756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:31.502792 (+ 36us) server_negotiation.cc:407] Connection header received
0923 15:15:31.503141 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:31.503147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:31.503200 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:31.503351 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:31.503568 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:31.504963 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:31.506057 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:31.506647 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:31.506789 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:31.507507 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:31.507511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:31.507511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:31.507744 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:31.507781 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:31.507797 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:31.507869 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:31.507936 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 15:15:31.507998 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260923 15:15:31.509123 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:31.513840 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:31.510523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33765 (local address 127.0.0.1:34797)
0923 15:15:31.510558 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:31.510560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:31.510741 (+ 181us) server_negotiation.cc:407] Connection header received
0923 15:15:31.510783 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:31.510784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:31.510794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:31.510859 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:31.510957 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:31.511690 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:31.512634 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:31.512941 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:31.513016 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:31.513607 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:31.513609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:31.513609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:31.513614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:31.513631 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:31.513636 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:31.513696 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:31.513747 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 15:15:31.513771 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 15:15:31.516332 10616 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:31.517056 10616 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:31.519742 10616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 560a3b769e034691b690503ed7836fdf: Generated new TSK 0
I20260923 15:15:31.519898 10616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:31.522104 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10535
2026-09-23T15:15:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:31 dist-test-slave-t5lt krb5kdc[10634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:31 dist-test-slave-t5lt krb5kdc[10634](info): set up 2 sockets
Sep 23 15:15:31 dist-test-slave-t5lt krb5kdc[10634](info): commencing operation
krb5kdc: starting...
W20260923 15:15:32.559234 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:32 dist-test-slave-t5lt krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:32.676311 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:32.755226 10647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:32.755463 10647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:32.755512 10647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:32.756664 10647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:32.756734 10647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:32.756762 10647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:32.756794 10647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:32.756816 10647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:32.758317 10647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:32.758602 10647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:32.758832 10647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:32.763788 10653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:32.763844 10652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:32.763861 10655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:32.763957 10647 server_base.cc:1034] running on GCE node
I20260923 15:15:32.764289 10647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:32.764612 10647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:32.765758 10647 hybrid_clock.cc:648] HybridClock initialized: now 1790176532765745 us; error 30 us; skew 500 ppm
Sep 23 15:15:32 dist-test-slave-t5lt krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:32.768185 10647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:32.768680 10647 webserver.cc:460] Webserver started at http://127.0.0.1:37983/ using document root <none> and password file <none>
I20260923 15:15:32.768923 10647 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:32.769001 10647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:32.769160 10647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:32.770303 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9"
format_stamp: "Formatted at 2026-09-23 15:15:32 on dist-test-slave-t5lt"
server_key: "526ba6aeabe64d6f283e0325bf818af6"
server_key_iv: "d8e9cb4bf5694418be3782f21dda8ddb"
server_key_version: "encryptionkey@0"
I20260923 15:15:32.770664 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9"
format_stamp: "Formatted at 2026-09-23 15:15:32 on dist-test-slave-t5lt"
server_key: "526ba6aeabe64d6f283e0325bf818af6"
server_key_iv: "d8e9cb4bf5694418be3782f21dda8ddb"
server_key_version: "encryptionkey@0"
I20260923 15:15:32.771998 10647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:32.772799 10662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:32.773123 10647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:32.773231 10647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9"
format_stamp: "Formatted at 2026-09-23 15:15:32 on dist-test-slave-t5lt"
server_key: "526ba6aeabe64d6f283e0325bf818af6"
server_key_iv: "d8e9cb4bf5694418be3782f21dda8ddb"
server_key_version: "encryptionkey@0"
I20260923 15:15:32.773306 10647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:32.776635 10647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:32.778678 10647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:32.778833 10647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:32.782891 10647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44709
I20260923 15:15:32.782927 10714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44709 every 8 connection(s)
I20260923 15:15:32.783257 10647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:32.784253 10715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:32.786695 10715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9: Bootstrap starting.
I20260923 15:15:32.787321 10715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:32.787632 10715 log.cc:824] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:32.788347 10715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9: No bootstrap required, opened a new log
I20260923 15:15:32.790156 10715 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260923 15:15:32.790323 10715 raft_consensus.cc:374] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:32.790349 10715 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12b64fdf6c04dd18ad5dab78f9f90c9, State: Initialized, Role: FOLLOWER
I20260923 15:15:32.790512 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10647
I20260923 15:15:32.790525 10715 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260923 15:15:32.790589 10715 raft_consensus.cc:388] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:32.790637 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:32.790658 10715 raft_consensus.cc:482] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:32.790707 10715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:32.790833 22321 external_mini_cluster.cc:1428] Setting key 78418c8481cc67450214290f95aba0dc
I20260923 15:15:32.791447 10715 raft_consensus.cc:504] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260923 15:15:32.791568 10715 leader_election.cc:302] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12b64fdf6c04dd18ad5dab78f9f90c9; no voters:
I20260923 15:15:32.791760 10715 leader_election.cc:288] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:32.791822 10720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:32.791946 10720 raft_consensus.cc:686] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [term 1 LEADER]: Becoming Leader. State: Replica: a12b64fdf6c04dd18ad5dab78f9f90c9, State: Running, Role: LEADER
I20260923 15:15:32.792043 10715 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:32.792095 10720 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } }
I20260923 15:15:32.792651 10721 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } } }
I20260923 15:15:32.792769 10721 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:32.793397 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:32.791354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36165 (local address 127.0.0.1:44709)
0923 15:15:32.791568 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:32.791577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:32.791609 (+ 32us) server_negotiation.cc:407] Connection header received
0923 15:15:32.791741 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:32.791748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:32.791782 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:32.791934 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:32.792685 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:32.792689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:32.792689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:32.792694 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:32.792748 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:32.792773 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:32.792908 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:32.793137 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 15:15:32.793188 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260923 15:15:32.792861 10722 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12b64fdf6c04dd18ad5dab78f9f90c9. Latest consensus state: current_term: 1 leader_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b64fdf6c04dd18ad5dab78f9f90c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44709 } } }
I20260923 15:15:32.793931 10722 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9 [sys.catalog]: This master's current role is: LEADER
W20260923 15:15:32.794474 10735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:15:32.794529 10735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:32.794512 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:32.794585 10736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:32.794708 10736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:32.796484 10736 catalog_manager.cc:1347] Generated new cluster ID: 56041e94db324ef58c9d0b65b071cdee
I20260923 15:15:32.796545 10736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:32.803083 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:32.796097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34647 (local address 127.0.0.1:44709)
0923 15:15:32.796131 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:32.796133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:32.802086 (+ 5953us) server_negotiation.cc:407] Connection header received
0923 15:15:32.802167 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:32.802168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:32.802174 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:32.802220 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:32.802846 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:32.802847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:32.802848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:32.802852 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:32.802867 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:32.802871 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:32.802922 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:32.802975 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 15:15:32.802989 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 15:15:32.819164 10736 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:32.819355 10736 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:32.825534 10736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a12b64fdf6c04dd18ad5dab78f9f90c9: Generated new TSK 0
I20260923 15:15:32.825718 10736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:32.833009 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10647
2026-09-23T15:15:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:32 dist-test-slave-t5lt krb5kdc[10746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:32 dist-test-slave-t5lt krb5kdc[10746](info): set up 2 sockets
Sep 23 15:15:32 dist-test-slave-t5lt krb5kdc[10746](info): commencing operation
krb5kdc: starting...
W20260923 15:15:33.875257 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:33 dist-test-slave-t5lt krb5kdc[10746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:33.997362 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:34.077601 10759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:34.077806 10759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:34.077831 10759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:34.079018 10759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:34.079066 10759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:34.079080 10759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:34.079092 10759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:34.079105 10759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:34.080606 10759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:34.080859 10759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:34.081099 10759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:15:34.086040 10759 server_base.cc:1034] running on GCE node
W20260923 15:15:34.086159 10764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:34.086174 10767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:34.086215 10765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:34.086555 10759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:34.086897 10759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:34.088063 10759 hybrid_clock.cc:648] HybridClock initialized: now 1790176534088056 us; error 37 us; skew 500 ppm
Sep 23 15:15:34 dist-test-slave-t5lt krb5kdc[10746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:34.090523 10759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:34.090951 10759 webserver.cc:460] Webserver started at http://127.0.0.1:42951/ using document root <none> and password file <none>
I20260923 15:15:34.091146 10759 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:34.091187 10759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:34.091272 10759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:34.092303 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "e66bb52eb28248329850286867eff492"
format_stamp: "Formatted at 2026-09-23 15:15:34 on dist-test-slave-t5lt"
server_key: "6a2f5aaf55e37f18ba9867543b44bde3"
server_key_iv: "6a96153c8119080d3d9446617b014a3e"
server_key_version: "encryptionkey@0"
I20260923 15:15:34.092612 10759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "e66bb52eb28248329850286867eff492"
format_stamp: "Formatted at 2026-09-23 15:15:34 on dist-test-slave-t5lt"
server_key: "6a2f5aaf55e37f18ba9867543b44bde3"
server_key_iv: "6a96153c8119080d3d9446617b014a3e"
server_key_version: "encryptionkey@0"
I20260923 15:15:34.093974 10759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:15:34.094696 10774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:34.094950 10759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:34.095055 10759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "e66bb52eb28248329850286867eff492"
format_stamp: "Formatted at 2026-09-23 15:15:34 on dist-test-slave-t5lt"
server_key: "6a2f5aaf55e37f18ba9867543b44bde3"
server_key_iv: "6a96153c8119080d3d9446617b014a3e"
server_key_version: "encryptionkey@0"
I20260923 15:15:34.095144 10759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:34.109580 10759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:34.111577 10759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:34.111739 10759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:34.115777 10759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36045
I20260923 15:15:34.115820 10826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36045 every 8 connection(s)
I20260923 15:15:34.116175 10759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:34.117080 10827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:34.119315 10827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492: Bootstrap starting.
I20260923 15:15:34.119840 10827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:34.120069 10827 log.cc:824] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:34.120735 10827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492: No bootstrap required, opened a new log
I20260923 15:15:34.122051 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10759
I20260923 15:15:34.122188 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:34.122388 22321 external_mini_cluster.cc:1428] Setting key 400570857fc9553290b24d7e116e97c9
I20260923 15:15:34.122561 10827 raft_consensus.cc:348] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66bb52eb28248329850286867eff492" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260923 15:15:34.122716 10827 raft_consensus.cc:374] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:34.122753 10827 raft_consensus.cc:729] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66bb52eb28248329850286867eff492, State: Initialized, Role: FOLLOWER
I20260923 15:15:34.122901 10827 consensus_queue.cc:260] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66bb52eb28248329850286867eff492" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260923 15:15:34.123000 10827 raft_consensus.cc:388] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:34.123041 10827 raft_consensus.cc:482] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:34.123121 10827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:34.124318 10827 raft_consensus.cc:504] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66bb52eb28248329850286867eff492" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260923 15:15:34.124449 10827 leader_election.cc:302] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e66bb52eb28248329850286867eff492; no voters:
I20260923 15:15:34.124624 10827 leader_election.cc:288] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:34.124665 10832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:34.124817 10832 raft_consensus.cc:686] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [term 1 LEADER]: Becoming Leader. State: Replica: e66bb52eb28248329850286867eff492, State: Running, Role: LEADER
I20260923 15:15:34.124886 10827 sys_catalog.cc:564] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:34.124950 10832 consensus_queue.cc:237] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66bb52eb28248329850286867eff492" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } }
I20260923 15:15:34.125312 10832 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e66bb52eb28248329850286867eff492. Latest consensus state: current_term: 1 leader_uuid: "e66bb52eb28248329850286867eff492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66bb52eb28248329850286867eff492" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } } }
I20260923 15:15:34.125415 10832 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:34.125322 10833 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e66bb52eb28248329850286867eff492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66bb52eb28248329850286867eff492" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36045 } } }
I20260923 15:15:34.125545 10833 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:34.125646 10840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:34.126186 10840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:34.127918 10840 catalog_manager.cc:1347] Generated new cluster ID: ea157fe7caaa48859cea94cb12588b2c
I20260923 15:15:34.127975 10840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:34 dist-test-slave-t5lt krb5kdc[10746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 15:15:34.133231 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:34.122921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:42229 (local address 127.0.0.1:36045)
0923 15:15:34.123136 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:34.123145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:34.123180 (+ 35us) server_negotiation.cc:407] Connection header received
0923 15:15:34.123308 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:34.123314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:34.123358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:34.123486 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:34.124329 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:34.125655 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:34.126815 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:34.127381 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:34.127516 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:34.129938 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:34.129952 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:34.129956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:34.129967 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:34.131611 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:34.132012 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:34.132015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:34.132015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:34.132114 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:34.132356 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:34.132357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:34.132357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:34.132597 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:34.132788 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:34.132881 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 15:15:34.133091 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
I20260923 15:15:34.134169 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:34.137521 10840 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:34.138206 10840 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:34.142411 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:34.135573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:47497 (local address 127.0.0.1:36045)
0923 15:15:34.135610 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:34.135612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:34.135731 (+ 119us) server_negotiation.cc:407] Connection header received
0923 15:15:34.135814 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:34.135815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:34.135824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:34.135877 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:34.136370 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:34.136868 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:34.138007 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:34.138311 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:34.138357 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:34.139699 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:34.139700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:34.139700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:34.139706 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:34.141225 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:34.141605 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:34.141606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:34.141606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:34.141652 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:34.141837 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:34.141838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:34.141838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:34.141976 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:34.142058 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:34.142264 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 15:15:34.142328 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:34.143641 10840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e66bb52eb28248329850286867eff492: Generated new TSK 0
I20260923 15:15:34.143818 10840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:34.150211 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10759
W20260923 15:15:34.154801 10744 connection.cc:535] client connection to 127.0.0.1:36045 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36045): Connection reset by peer (error 104)
2026-09-23T15:15:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:34 dist-test-slave-t5lt krb5kdc[10858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:34 dist-test-slave-t5lt krb5kdc[10858](info): set up 2 sockets
Sep 23 15:15:34 dist-test-slave-t5lt krb5kdc[10858](info): commencing operation
krb5kdc: starting...
W20260923 15:15:35.179328 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:35 dist-test-slave-t5lt krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:35.297278 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:35.377549 10871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:35.377779 10871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:35.377816 10871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:35.379159 10871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:35.379215 10871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:35.379240 10871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:35.379259 10871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:35.379281 10871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:35.380972 10871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:35.381165 10871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:35.381321 10871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:35.386104 10877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:35.386188 10871 server_base.cc:1034] running on GCE node
W20260923 15:15:35.386104 10876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:35.386132 10879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:35.386541 10871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:35.386829 10871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:35.388018 10871 hybrid_clock.cc:648] HybridClock initialized: now 1790176535388006 us; error 26 us; skew 500 ppm
Sep 23 15:15:35 dist-test-slave-t5lt krb5kdc[10858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:35.390394 10871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:35.390810 10871 webserver.cc:460] Webserver started at http://127.0.0.1:46445/ using document root <none> and password file <none>
I20260923 15:15:35.391058 10871 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:35.391129 10871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:35.391278 10871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:35.392338 10871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "7f37f9e24b0b43b4bc3bd012023bb468"
format_stamp: "Formatted at 2026-09-23 15:15:35 on dist-test-slave-t5lt"
server_key: "a9b85f1d08b6bd03d89bd1487ef7c112"
server_key_iv: "7a994fd1c43b3b7574d08dfb4607ebe9"
server_key_version: "encryptionkey@0"
I20260923 15:15:35.392705 10871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "7f37f9e24b0b43b4bc3bd012023bb468"
format_stamp: "Formatted at 2026-09-23 15:15:35 on dist-test-slave-t5lt"
server_key: "a9b85f1d08b6bd03d89bd1487ef7c112"
server_key_iv: "7a994fd1c43b3b7574d08dfb4607ebe9"
server_key_version: "encryptionkey@0"
I20260923 15:15:35.394147 10871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:35.394995 10886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:35.395202 10871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:35.395308 10871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "7f37f9e24b0b43b4bc3bd012023bb468"
format_stamp: "Formatted at 2026-09-23 15:15:35 on dist-test-slave-t5lt"
server_key: "a9b85f1d08b6bd03d89bd1487ef7c112"
server_key_iv: "7a994fd1c43b3b7574d08dfb4607ebe9"
server_key_version: "encryptionkey@0"
I20260923 15:15:35.395401 10871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:35.405005 10871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:35.407112 10871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:35.407263 10871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:35.411273 10871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45517
I20260923 15:15:35.411595 10938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45517 every 8 connection(s)
I20260923 15:15:35.411653 10871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:35.412125 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10871
I20260923 15:15:35.412335 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:35.412485 10939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:35.412570 22321 external_mini_cluster.cc:1428] Setting key 83927537229c9729f2b1fb6254ddeb38
I20260923 15:15:35.416321 10939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468: Bootstrap starting.
I20260923 15:15:35.417030 10939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:35.417320 10939 log.cc:824] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:35.418150 10939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468: No bootstrap required, opened a new log
I20260923 15:15:35.419127 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:35.413123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:36249 (local address 127.0.0.1:45517)
0923 15:15:35.413389 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:35.413399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:35.413431 (+ 32us) server_negotiation.cc:407] Connection header received
0923 15:15:35.414219 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:35.414228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:35.414307 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:35.414466 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:35.414640 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:35.416121 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:35.416937 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:35.417576 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:35.417662 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:35.418439 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:35.418443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:35.418444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:35.418647 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:35.418687 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:35.418703 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:35.418797 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:35.418879 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 15:15:35.418906 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":67,"threads_started":1}
I20260923 15:15:35.420269 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:35.420601 10939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260923 15:15:35.420737 10939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:35.420758 10939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f37f9e24b0b43b4bc3bd012023bb468, State: Initialized, Role: FOLLOWER
I20260923 15:15:35.420897 10939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260923 15:15:35.420969 10939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:35.420991 10939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:35.421059 10939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:35.421900 10939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260923 15:15:35.422031 10939 leader_election.cc:302] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f37f9e24b0b43b4bc3bd012023bb468; no voters:
I20260923 15:15:35.422186 10939 leader_election.cc:288] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:35.422217 10949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:35.422376 10949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [term 1 LEADER]: Becoming Leader. State: Replica: 7f37f9e24b0b43b4bc3bd012023bb468, State: Running, Role: LEADER
I20260923 15:15:35.422492 10939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:35.422508 10949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260923 15:15:35.422971 10949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f37f9e24b0b43b4bc3bd012023bb468. Latest consensus state: current_term: 1 leader_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } } }
I20260923 15:15:35.423067 10949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:35.423143 10950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f37f9e24b0b43b4bc3bd012023bb468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } } }
I20260923 15:15:35.423301 10950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:35.423758 10958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:35.424237 10958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:35.425392 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:35.421787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:46867 (local address 127.0.0.1:45517)
0923 15:15:35.421832 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:35.421834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:35.421918 (+ 84us) server_negotiation.cc:407] Connection header received
0923 15:15:35.422015 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:35.422016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:35.422025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:35.422074 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:35.422226 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:35.422888 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:35.423984 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:35.424296 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:35.424369 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:35.425030 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:35.425031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:35.425032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:35.425036 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:35.425049 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:35.425054 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:35.425100 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:35.425310 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 15:15:35.425323 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:15:35.426501 10958 catalog_manager.cc:1347] Generated new cluster ID: 2c9cb6cc2e8842eba3570a5ead8960d9
I20260923 15:15:35.426540 10958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:35.442924 10958 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:35.443440 10958 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:35.449692 10958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f37f9e24b0b43b4bc3bd012023bb468: Generated new TSK 0
I20260923 15:15:35.449891 10958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:35.455076 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10871
W20260923 15:15:35.459828 10856 connection.cc:535] client connection to 127.0.0.1:45517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45517): Connection reset by peer (error 104)
2026-09-23T15:15:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:35 dist-test-slave-t5lt krb5kdc[10970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:35 dist-test-slave-t5lt krb5kdc[10970](info): set up 2 sockets
Sep 23 15:15:35 dist-test-slave-t5lt krb5kdc[10970](info): commencing operation
krb5kdc: starting...
W20260923 15:15:36.499365 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:36 dist-test-slave-t5lt krb5kdc[10970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:36.613042 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:36.692863 10983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:36.693107 10983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:36.693146 10983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:36.694350 10983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:36.694423 10983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:36.694448 10983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:36.694478 10983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:36.694509 10983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:36.696003 10983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:36.696280 10983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:36.696506 10983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:36.701360 10991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:36.701447 10989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:36.701360 10988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:36.701578 10983 server_base.cc:1034] running on GCE node
I20260923 15:15:36.701817 10983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:36.702121 10983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:36.703286 10983 hybrid_clock.cc:648] HybridClock initialized: now 1790176536703267 us; error 35 us; skew 500 ppm
Sep 23 15:15:36 dist-test-slave-t5lt krb5kdc[10970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:36.711194 10983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:36.711642 10983 webserver.cc:460] Webserver started at http://127.0.0.1:39061/ using document root <none> and password file <none>
I20260923 15:15:36.711871 10983 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:36.711946 10983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:36.712107 10983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:36.713168 10983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc"
format_stamp: "Formatted at 2026-09-23 15:15:36 on dist-test-slave-t5lt"
server_key: "bdfc294e357274c402598910018797b2"
server_key_iv: "6c6f073a7bb4ae87877f5d058fe15e55"
server_key_version: "encryptionkey@0"
I20260923 15:15:36.713560 10983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc"
format_stamp: "Formatted at 2026-09-23 15:15:36 on dist-test-slave-t5lt"
server_key: "bdfc294e357274c402598910018797b2"
server_key_iv: "6c6f073a7bb4ae87877f5d058fe15e55"
server_key_version: "encryptionkey@0"
I20260923 15:15:36.715037 10983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:36.715947 10998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:36.716403 10983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:36.716542 10983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc"
format_stamp: "Formatted at 2026-09-23 15:15:36 on dist-test-slave-t5lt"
server_key: "bdfc294e357274c402598910018797b2"
server_key_iv: "6c6f073a7bb4ae87877f5d058fe15e55"
server_key_version: "encryptionkey@0"
I20260923 15:15:36.716624 10983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:36.720083 10983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:36.722172 10983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:36.722328 10983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:36.726352 10983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42741
I20260923 15:15:36.726375 11050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42741 every 8 connection(s)
I20260923 15:15:36.726701 10983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:36.727509 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 10983
I20260923 15:15:36.727644 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:36.727685 11051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:36.727838 22321 external_mini_cluster.cc:1428] Setting key 97d603641f585eee2873a33a2badbd98
I20260923 15:15:36.730377 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:36.728377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:47203 (local address 127.0.0.1:42741)
0923 15:15:36.728552 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:36.728559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:36.728583 (+ 24us) server_negotiation.cc:407] Connection header received
0923 15:15:36.729083 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:36.729089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:36.729120 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:36.729231 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:36.729915 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:36.729919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:36.729919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:36.729924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:36.729966 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:36.729977 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:36.730044 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:36.730130 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 15:15:36.730180 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260923 15:15:36.731002 11051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc: Bootstrap starting.
I20260923 15:15:36.731249 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:36.731662 11051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:36.731963 11051 log.cc:824] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:36.732687 11051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc: No bootstrap required, opened a new log
I20260923 15:15:36.733600 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:36.732697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:56189 (local address 127.0.0.1:42741)
0923 15:15:36.732737 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:36.732739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:36.732830 (+ 91us) server_negotiation.cc:407] Connection header received
0923 15:15:36.732908 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:36.732909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:36.732913 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:36.732958 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:36.733393 (+ 435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:36.733394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:36.733394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:36.733397 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:36.733407 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:36.733410 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:36.733445 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:36.733511 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 15:15:36.733520 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:15:36.734954 11051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260923 15:15:36.735082 11051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:36.735105 11051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd5c5a8b0614e42b9d4bb8153ac42fc, State: Initialized, Role: FOLLOWER
I20260923 15:15:36.735217 11051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260923 15:15:36.735299 11051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:36.735321 11051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:36.735349 11051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:36.735810 11051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260923 15:15:36.735901 11051 leader_election.cc:302] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dd5c5a8b0614e42b9d4bb8153ac42fc; no voters:
I20260923 15:15:36.736044 11051 leader_election.cc:288] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:36.736124 11061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:36.736276 11061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [term 1 LEADER]: Becoming Leader. State: Replica: 3dd5c5a8b0614e42b9d4bb8153ac42fc, State: Running, Role: LEADER
I20260923 15:15:36.736343 11051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:36.736402 11061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } }
I20260923 15:15:36.736788 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dd5c5a8b0614e42b9d4bb8153ac42fc. Latest consensus state: current_term: 1 leader_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } } }
I20260923 15:15:36.736889 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:36.737097 11062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd5c5a8b0614e42b9d4bb8153ac42fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42741 } } }
I20260923 15:15:36.737169 11062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:36.737336 11069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:36.737864 11069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:36.739364 11069 catalog_manager.cc:1347] Generated new cluster ID: ee2dd07ad1c14e549732ce6556410406
I20260923 15:15:36.739423 11069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:36.761543 11069 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:36.761722 11069 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:36.775273 11069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dd5c5a8b0614e42b9d4bb8153ac42fc: Generated new TSK 0
I20260923 15:15:36.775477 11069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:36.781114 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 10983
2026-09-23T15:15:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:36 dist-test-slave-t5lt krb5kdc[11082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:36 dist-test-slave-t5lt krb5kdc[11082](info): set up 2 sockets
Sep 23 15:15:36 dist-test-slave-t5lt krb5kdc[11082](info): commencing operation
krb5kdc: starting...
W20260923 15:15:37.811389 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:37 dist-test-slave-t5lt krb5kdc[11082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:37.928478 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:38.008893 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:38.009099 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:38.009121 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:38.010329 11095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:38.010373 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:38.010388 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:38.010401 11095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:38.010442 11095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:38.011910 11095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:38.012164 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:38.012408 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:38.017388 11100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:38.017405 11101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:38.017355 11103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:38.017590 11095 server_base.cc:1034] running on GCE node
I20260923 15:15:38.017840 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:38.018158 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:38.019372 11095 hybrid_clock.cc:648] HybridClock initialized: now 1790176538019353 us; error 48 us; skew 500 ppm
Sep 23 15:15:38 dist-test-slave-t5lt krb5kdc[11082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:38.022133 11095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:38.022604 11095 webserver.cc:460] Webserver started at http://127.0.0.1:35159/ using document root <none> and password file <none>
I20260923 15:15:38.022866 11095 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:38.022946 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:38.023098 11095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:38.024133 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "95bb603e556b48c4be12db2f7a0458dd"
format_stamp: "Formatted at 2026-09-23 15:15:38 on dist-test-slave-t5lt"
server_key: "777a4ee2600ded1263822b641800f3f2"
server_key_iv: "ee472646ddbec300f4ae221c5ee73a72"
server_key_version: "encryptionkey@0"
I20260923 15:15:38.024502 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "95bb603e556b48c4be12db2f7a0458dd"
format_stamp: "Formatted at 2026-09-23 15:15:38 on dist-test-slave-t5lt"
server_key: "777a4ee2600ded1263822b641800f3f2"
server_key_iv: "ee472646ddbec300f4ae221c5ee73a72"
server_key_version: "encryptionkey@0"
I20260923 15:15:38.026106 11095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:38.026929 11110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:38.027129 11095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:38.027225 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "95bb603e556b48c4be12db2f7a0458dd"
format_stamp: "Formatted at 2026-09-23 15:15:38 on dist-test-slave-t5lt"
server_key: "777a4ee2600ded1263822b641800f3f2"
server_key_iv: "ee472646ddbec300f4ae221c5ee73a72"
server_key_version: "encryptionkey@0"
I20260923 15:15:38.027308 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:38.047673 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:38.049688 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:38.049860 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:38.053814 11095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40663
I20260923 15:15:38.053870 11162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40663 every 8 connection(s)
I20260923 15:15:38.054267 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:38.055128 11163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:38.057492 11163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd: Bootstrap starting.
I20260923 15:15:38.058051 11163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:38.058288 11163 log.cc:824] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:38.059026 11163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd: No bootstrap required, opened a new log
I20260923 15:15:38.060606 11163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bb603e556b48c4be12db2f7a0458dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260923 15:15:38.060745 11163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:38.060768 11163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bb603e556b48c4be12db2f7a0458dd, State: Initialized, Role: FOLLOWER
I20260923 15:15:38.060878 11163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bb603e556b48c4be12db2f7a0458dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260923 15:15:38.060973 11163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:38.061014 11163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:38.061091 11163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:38.061640 11163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bb603e556b48c4be12db2f7a0458dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260923 15:15:38.061762 11163 leader_election.cc:302] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95bb603e556b48c4be12db2f7a0458dd; no voters:
I20260923 15:15:38.061977 11163 leader_election.cc:288] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:38.062028 11166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:38.062209 11166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [term 1 LEADER]: Becoming Leader. State: Replica: 95bb603e556b48c4be12db2f7a0458dd, State: Running, Role: LEADER
I20260923 15:15:38.062260 11163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:38.062362 11166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bb603e556b48c4be12db2f7a0458dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } }
I20260923 15:15:38.062732 11168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95bb603e556b48c4be12db2f7a0458dd. Latest consensus state: current_term: 1 leader_uuid: "95bb603e556b48c4be12db2f7a0458dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bb603e556b48c4be12db2f7a0458dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20260923 15:15:38.062882 11168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:38.062883 11167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95bb603e556b48c4be12db2f7a0458dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bb603e556b48c4be12db2f7a0458dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40663 } } }
I20260923 15:15:38.063009 11167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:38.063297 11175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:38.063527 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11095
I20260923 15:15:38.063653 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:38.063861 22321 external_mini_cluster.cc:1428] Setting key 5d5064c84a27c73849a8014e322ad9d8
I20260923 15:15:38.063874 11175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:38.065778 11175 catalog_manager.cc:1347] Generated new cluster ID: 26cec995981d4b1da80a8c33496270bf
I20260923 15:15:38.065835 11175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:38 dist-test-slave-t5lt krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 15:15:38.075933 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:38.064351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:44887 (local address 127.0.0.1:40663)
0923 15:15:38.064791 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:38.064801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:38.064830 (+ 29us) server_negotiation.cc:407] Connection header received
0923 15:15:38.064942 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:38.064948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:38.065004 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:38.065146 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:38.065961 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:38.067351 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:38.068394 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:38.068995 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:38.069082 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:38.071967 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:38.071972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:38.071972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:38.071984 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:38.073955 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:38.074379 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:38.074381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:38.074382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:38.074482 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:38.074693 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:38.074693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:38.074694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:38.075001 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:38.075139 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:38.075597 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 15:15:38.075695 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":76,"threads_started":1}
I20260923 15:15:38.076654 11175 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:38.076962 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:38.077291 11175 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:38.082962 11175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95bb603e556b48c4be12db2f7a0458dd: Generated new TSK 0
I20260923 15:15:38.083142 11175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:38.088788 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:38.078451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:38505 (local address 127.0.0.1:40663)
0923 15:15:38.078485 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:38.078487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:38.082057 (+ 3570us) server_negotiation.cc:407] Connection header received
0923 15:15:38.082120 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:38.082121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:38.082131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:38.082192 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:38.082880 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:38.083727 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:38.084776 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:38.084994 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:38.085047 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:38.086403 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:38.086404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:38.086405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:38.086411 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:38.087667 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:38.088067 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:38.088067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:38.088068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:38.088144 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:38.088369 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:38.088370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:38.088370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:38.088459 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:38.088537 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:38.088688 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 15:15:38.088732 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 15:15:38.090927 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11095
W20260923 15:15:38.095297 11080 connection.cc:535] client connection to 127.0.0.1:40663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40663): Connection reset by peer (error 104)
2026-09-23T15:15:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:38 dist-test-slave-t5lt krb5kdc[11194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:38 dist-test-slave-t5lt krb5kdc[11194](info): set up 2 sockets
Sep 23 15:15:38 dist-test-slave-t5lt krb5kdc[11194](info): commencing operation
krb5kdc: starting...
W20260923 15:15:39.119288 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:39 dist-test-slave-t5lt krb5kdc[11194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:39.235059 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:39.316505 11207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:39.316751 11207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:39.316787 11207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:39.318037 11207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:39.318099 11207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:39.318122 11207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:39.318149 11207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:39.318181 11207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:39.319679 11207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:39.319958 11207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:39.320189 11207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:39.325048 11213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:39.325067 11212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:39.325268 11207 server_base.cc:1034] running on GCE node
W20260923 15:15:39.325275 11215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:39.325604 11207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:39.325934 11207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:39.327116 11207 hybrid_clock.cc:648] HybridClock initialized: now 1790176539327105 us; error 40 us; skew 500 ppm
Sep 23 15:15:39 dist-test-slave-t5lt krb5kdc[11194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:39.329586 11207 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:39.330018 11207 webserver.cc:460] Webserver started at http://127.0.0.1:37357/ using document root <none> and password file <none>
I20260923 15:15:39.330210 11207 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:39.330250 11207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:39.330333 11207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:39.331316 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "5a189a07132c47599e38fba2c4324fde"
format_stamp: "Formatted at 2026-09-23 15:15:39 on dist-test-slave-t5lt"
server_key: "4062adac17f39faf09cb990a2275d4e7"
server_key_iv: "80f0c376003a759ada2df13256ef7cc4"
server_key_version: "encryptionkey@0"
I20260923 15:15:39.331614 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "5a189a07132c47599e38fba2c4324fde"
format_stamp: "Formatted at 2026-09-23 15:15:39 on dist-test-slave-t5lt"
server_key: "4062adac17f39faf09cb990a2275d4e7"
server_key_iv: "80f0c376003a759ada2df13256ef7cc4"
server_key_version: "encryptionkey@0"
I20260923 15:15:39.332886 11207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 15:15:39.333562 11222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:39.333729 11207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:15:39.333815 11207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "5a189a07132c47599e38fba2c4324fde"
format_stamp: "Formatted at 2026-09-23 15:15:39 on dist-test-slave-t5lt"
server_key: "4062adac17f39faf09cb990a2275d4e7"
server_key_iv: "80f0c376003a759ada2df13256ef7cc4"
server_key_version: "encryptionkey@0"
I20260923 15:15:39.333938 11207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:39.349203 11207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:39.351238 11207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:39.351404 11207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:39.355257 11207 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44609
I20260923 15:15:39.355293 11274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44609 every 8 connection(s)
I20260923 15:15:39.355646 11207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:39.356521 11275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:39.358808 11275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde: Bootstrap starting.
I20260923 15:15:39.359444 11275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:39.359493 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11207
I20260923 15:15:39.359617 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:39.359838 22321 external_mini_cluster.cc:1428] Setting key 6a4887863dd9b58523e1b320085ffecd
I20260923 15:15:39.359915 11275 log.cc:824] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:39.360805 11275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde: No bootstrap required, opened a new log
I20260923 15:15:39.363390 11275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a189a07132c47599e38fba2c4324fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44609 } }
I20260923 15:15:39.363525 11275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:39.363547 11275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a189a07132c47599e38fba2c4324fde, State: Initialized, Role: FOLLOWER
I20260923 15:15:39.363686 11275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a189a07132c47599e38fba2c4324fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44609 } }
I20260923 15:15:39.363754 11275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:39.363775 11275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:39.363842 11275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:39.364434 11275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a189a07132c47599e38fba2c4324fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44609 } }
I20260923 15:15:39.364562 11275 leader_election.cc:302] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a189a07132c47599e38fba2c4324fde; no voters:
I20260923 15:15:39.364785 11275 leader_election.cc:288] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:39.364975 11280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:39.365113 11280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [term 1 LEADER]: Becoming Leader. State: Replica: 5a189a07132c47599e38fba2c4324fde, State: Running, Role: LEADER
I20260923 15:15:39.365201 11280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a189a07132c47599e38fba2c4324fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44609 } }
I20260923 15:15:39.365670 11282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a189a07132c47599e38fba2c4324fde. Latest consensus state: current_term: 1 leader_uuid: "5a189a07132c47599e38fba2c4324fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a189a07132c47599e38fba2c4324fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44609 } } }
I20260923 15:15:39.365675 11281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a189a07132c47599e38fba2c4324fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a189a07132c47599e38fba2c4324fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44609 } } }
I20260923 15:15:39.365770 11282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:39.365234 11275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:39.365792 11281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:39.366034 11284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:39.366556 11284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:39.367486 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:39.360336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:32887 (local address 127.0.0.1:44609)
0923 15:15:39.360524 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:39.360533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:39.360566 (+ 33us) server_negotiation.cc:407] Connection header received
0923 15:15:39.361190 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:39.361199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:39.361264 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:39.361425 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:39.361573 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:39.362991 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:39.363958 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:39.364365 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:39.364519 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:39.365322 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:39.365326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:39.365327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:39.365544 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:39.365589 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:39.365605 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:39.365687 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:39.367280 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0923 15:15:39.367310 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260923 15:15:39.368233 11284 catalog_manager.cc:1347] Generated new cluster ID: a0043fab92b14f76a89ddb941a87b207
I20260923 15:15:39.368312 11284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:39.368642 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:39.372375 11284 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:39.373087 11284 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:39.373983 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:39.370213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:37759 (local address 127.0.0.1:44609)
0923 15:15:39.370252 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:39.370253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:39.370398 (+ 145us) server_negotiation.cc:407] Connection header received
0923 15:15:39.370428 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:39.370429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:39.370440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:39.370495 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:39.370644 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:39.371566 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:39.372603 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:39.372911 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:39.372984 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:39.373705 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:39.373706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:39.373707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:39.373711 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:39.373727 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:39.373731 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:39.373786 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:39.373898 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 15:15:39.373909 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":21}
I20260923 15:15:39.387713 11284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a189a07132c47599e38fba2c4324fde: Generated new TSK 0
I20260923 15:15:39.387972 11284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:39.398200 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11207
W20260923 15:15:39.402997 11192 connection.cc:535] client connection to 127.0.0.1:44609 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44609): Connection reset by peer (error 104)
2026-09-23T15:15:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:39 dist-test-slave-t5lt krb5kdc[11306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:39 dist-test-slave-t5lt krb5kdc[11306](info): set up 2 sockets
Sep 23 15:15:39 dist-test-slave-t5lt krb5kdc[11306](info): commencing operation
krb5kdc: starting...
W20260923 15:15:40.439672 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:40 dist-test-slave-t5lt krb5kdc[11306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 15:15:40.556577 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:40.635342 11319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:40.635576 11319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:40.635613 11319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:40.636720 11319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:40.636786 11319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:40.636811 11319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:40.636839 11319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:40.636871 11319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:40.638317 11319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:40.638589 11319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:40.638806 11319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:40.643815 11327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:40.643931 11319 server_base.cc:1034] running on GCE node
W20260923 15:15:40.643797 11324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:40.643790 11325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:40.644228 11319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:40.644526 11319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:40.645669 11319 hybrid_clock.cc:648] HybridClock initialized: now 1790176540645649 us; error 37 us; skew 500 ppm
Sep 23 15:15:40 dist-test-slave-t5lt krb5kdc[11306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:40.648048 11319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 15:15:40.648504 11319 webserver.cc:460] Webserver started at http://127.0.0.1:39045/ using document root <none> and password file <none>
I20260923 15:15:40.648720 11319 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:40.648787 11319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:40.648936 11319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:40.650023 11319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "8549a6ae8cb2431bb8c678d932514f4f"
format_stamp: "Formatted at 2026-09-23 15:15:40 on dist-test-slave-t5lt"
server_key: "eaedcd6808d564a11a0d1731d30d3fce"
server_key_iv: "ac5dd7401266b5c4a9cb839a70845ee6"
server_key_version: "encryptionkey@0"
I20260923 15:15:40.650382 11319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "8549a6ae8cb2431bb8c678d932514f4f"
format_stamp: "Formatted at 2026-09-23 15:15:40 on dist-test-slave-t5lt"
server_key: "eaedcd6808d564a11a0d1731d30d3fce"
server_key_iv: "ac5dd7401266b5c4a9cb839a70845ee6"
server_key_version: "encryptionkey@0"
I20260923 15:15:40.651701 11319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:40.652489 11334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:40.652688 11319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:40.652792 11319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "8549a6ae8cb2431bb8c678d932514f4f"
format_stamp: "Formatted at 2026-09-23 15:15:40 on dist-test-slave-t5lt"
server_key: "eaedcd6808d564a11a0d1731d30d3fce"
server_key_iv: "ac5dd7401266b5c4a9cb839a70845ee6"
server_key_version: "encryptionkey@0"
I20260923 15:15:40.652865 11319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:40.656167 11319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:40.658061 11319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:40.658209 11319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:40.662233 11319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36071
I20260923 15:15:40.662266 11386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36071 every 8 connection(s)
I20260923 15:15:40.662652 11319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:40.663580 11387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:40.665630 11387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f: Bootstrap starting.
I20260923 15:15:40.666195 11387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:40.666431 11387 log.cc:824] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:40.667042 11387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f: No bootstrap required, opened a new log
I20260923 15:15:40.668498 11387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8549a6ae8cb2431bb8c678d932514f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36071 } }
I20260923 15:15:40.668622 11387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:40.668643 11387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8549a6ae8cb2431bb8c678d932514f4f, State: Initialized, Role: FOLLOWER
I20260923 15:15:40.668743 11387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8549a6ae8cb2431bb8c678d932514f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36071 } }
I20260923 15:15:40.668792 11387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:40.668812 11387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:40.668841 11387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:40.669301 11387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8549a6ae8cb2431bb8c678d932514f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36071 } }
I20260923 15:15:40.669380 11387 leader_election.cc:302] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8549a6ae8cb2431bb8c678d932514f4f; no voters:
I20260923 15:15:40.669490 11387 leader_election.cc:288] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:40.669559 11390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:40.669731 11390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [term 1 LEADER]: Becoming Leader. State: Replica: 8549a6ae8cb2431bb8c678d932514f4f, State: Running, Role: LEADER
I20260923 15:15:40.669749 11387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:40.669816 11390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8549a6ae8cb2431bb8c678d932514f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36071 } }
I20260923 15:15:40.670234 11391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8549a6ae8cb2431bb8c678d932514f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8549a6ae8cb2431bb8c678d932514f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36071 } } }
I20260923 15:15:40.670264 11392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8549a6ae8cb2431bb8c678d932514f4f. Latest consensus state: current_term: 1 leader_uuid: "8549a6ae8cb2431bb8c678d932514f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8549a6ae8cb2431bb8c678d932514f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36071 } } }
I20260923 15:15:40.670483 11392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:40.670680 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11319
I20260923 15:15:40.670827 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:40.670905 11396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:40.671032 22321 external_mini_cluster.cc:1428] Setting key c0c7e74222ff4e8b30273d1bf92715e4
I20260923 15:15:40.671386 11396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:40.670392 11391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:40.673207 11396 catalog_manager.cc:1347] Generated new cluster ID: 35c49b0e001546d0b99d3b28b6e6e070
I20260923 15:15:40.673305 11396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:40.673489 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:40.671711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:47073 (local address 127.0.0.1:36071)
0923 15:15:40.672053 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:40.672061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:40.672093 (+ 32us) server_negotiation.cc:407] Connection header received
0923 15:15:40.672184 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:40.672189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:40.672211 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:40.672350 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:40.673009 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:40.673013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:40.673013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:40.673018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:40.673054 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:40.673069 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:40.673106 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:40.673242 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 15:15:40.673284 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":63,"threads_started":1}
I20260923 15:15:40.674415 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:40.680790 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:40.675744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.211:58387 (local address 127.0.0.1:36071)
0923 15:15:40.675784 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:40.675786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:40.679792 (+ 4006us) server_negotiation.cc:407] Connection header received
0923 15:15:40.679869 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:40.679870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:40.679876 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:40.679930 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:40.680554 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:40.680556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:40.680556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:40.680561 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:40.680586 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:40.680590 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:40.680646 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:40.680697 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 15:15:40.680712 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:15:40.689947 11396 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:40.690135 11396 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:40.696163 11396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8549a6ae8cb2431bb8c678d932514f4f: Generated new TSK 0
I20260923 15:15:40.696336 11396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:40.700299 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11319
2026-09-23T15:15:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:40 dist-test-slave-t5lt krb5kdc[11418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:40 dist-test-slave-t5lt krb5kdc[11418](info): set up 2 sockets
Sep 23 15:15:40 dist-test-slave-t5lt krb5kdc[11418](info): commencing operation
krb5kdc: starting...
W20260923 15:15:41.731289 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:41 dist-test-slave-t5lt krb5kdc[11418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:41.848074 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41311
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:41311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:41.929817 11431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:41.930104 11431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:41.930130 11431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:41.931207 11431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:41.931241 11431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:41.931255 11431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:41.931268 11431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:41.931280 11431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:41.932631 11431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:41311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:41.932809 11431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:41.932966 11431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:41.937697 11437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:41.937731 11436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:41.937731 11439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:41.937963 11431 server_base.cc:1034] running on GCE node
I20260923 15:15:41.938184 11431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:41.938499 11431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:41.939700 11431 hybrid_clock.cc:648] HybridClock initialized: now 1790176541939684 us; error 34 us; skew 500 ppm
Sep 23 15:15:41 dist-test-slave-t5lt krb5kdc[11418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176541, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:41.947603 11431 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:41.948055 11431 webserver.cc:460] Webserver started at http://127.21.204.126:34317/ using document root <none> and password file <none>
I20260923 15:15:41.948285 11431 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:41.948331 11431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:41.948462 11431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:41.949517 11431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f"
format_stamp: "Formatted at 2026-09-23 15:15:41 on dist-test-slave-t5lt"
server_key: "f2f91e322c8bee85fd68cfea353e4d8a"
server_key_iv: "ad547aece8723bf8922baed7b2c81a5d"
server_key_version: "encryptionkey@0"
I20260923 15:15:41.949934 11431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f"
format_stamp: "Formatted at 2026-09-23 15:15:41 on dist-test-slave-t5lt"
server_key: "f2f91e322c8bee85fd68cfea353e4d8a"
server_key_iv: "ad547aece8723bf8922baed7b2c81a5d"
server_key_version: "encryptionkey@0"
I20260923 15:15:41.951303 11431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:41.952138 11446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:41.952396 11431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:41.952492 11431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f"
format_stamp: "Formatted at 2026-09-23 15:15:41 on dist-test-slave-t5lt"
server_key: "f2f91e322c8bee85fd68cfea353e4d8a"
server_key_iv: "ad547aece8723bf8922baed7b2c81a5d"
server_key_version: "encryptionkey@0"
I20260923 15:15:41.952569 11431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:41.967553 11431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:41.969555 11431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:41.969722 11431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:41.973958 11431 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:41311
I20260923 15:15:41.973976 11498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:41311 every 8 connection(s)
I20260923 15:15:41.974334 11431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:41.975185 11499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:41.977415 11499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f: Bootstrap starting.
I20260923 15:15:41.978016 11499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:41.978302 11499 log.cc:824] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:41.978994 11499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f: No bootstrap required, opened a new log
I20260923 15:15:41.980546 11499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41311 } }
I20260923 15:15:41.980679 11499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:41.980700 11499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1a5e3eb5574fcc8cc418ca695e5a3f, State: Initialized, Role: FOLLOWER
I20260923 15:15:41.980855 11499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41311 } }
I20260923 15:15:41.980927 11499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:41.980947 11499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:41.980975 11499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:41.981807 11499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41311 } }
I20260923 15:15:41.981952 11499 leader_election.cc:302] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a1a5e3eb5574fcc8cc418ca695e5a3f; no voters:
I20260923 15:15:41.982137 11499 leader_election.cc:288] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:41.982177 11502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:41.982343 11502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [term 1 LEADER]: Becoming Leader. State: Replica: 4a1a5e3eb5574fcc8cc418ca695e5a3f, State: Running, Role: LEADER
I20260923 15:15:41.982396 11499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:41.982522 11502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41311 } }
I20260923 15:15:41.982607 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11431
I20260923 15:15:41.982714 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:41.982920 22321 external_mini_cluster.cc:1428] Setting key d8d3341806a1c4afd742e5c01f1467a0
I20260923 15:15:41.982977 11504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a1a5e3eb5574fcc8cc418ca695e5a3f. Latest consensus state: current_term: 1 leader_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41311 } } }
I20260923 15:15:41.983045 11504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:41.983211 11503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1a5e3eb5574fcc8cc418ca695e5a3f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 41311 } } }
I20260923 15:15:41.983269 11503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:41.983662 11511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:41.984282 11511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:41.986316 11511 catalog_manager.cc:1347] Generated new cluster ID: 8a6d6906d75a4de99272c1760822fe37
I20260923 15:15:41.986378 11511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:41 dist-test-slave-t5lt krb5kdc[11418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:41.995289 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:41.983580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33539 (local address 127.21.204.126:41311)
0923 15:15:41.983890 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:41.983900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:41.984019 (+ 119us) server_negotiation.cc:407] Connection header received
0923 15:15:41.984165 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:41.984173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:41.984232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:41.984376 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:41.985177 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:41.986245 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:41.987318 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:41.987912 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:41.988010 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:41.990848 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:41.990853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:41.990853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:41.990866 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:41.993483 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:41.993969 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:41.993972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:41.993972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:41.994089 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:41.994340 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:41.994341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:41.994341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:41.994651 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:41.994791 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:41.994953 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 15:15:41.995053 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":75,"threads_started":1}
I20260923 15:15:41.996270 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:42.000358 11511 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:42.000898 11511 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:42.005228 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:41.997539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46203 (local address 127.21.204.126:41311)
0923 15:15:41.997575 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:41.997578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:41.998025 (+ 447us) server_negotiation.cc:407] Connection header received
0923 15:15:41.998072 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:41.998073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:41.998084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:41.998150 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:41.998704 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:41.999589 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:42.000502 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:42.000833 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:42.000902 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:42.002218 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:42.002219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:42.002220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:42.002227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:42.003881 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:42.004266 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:42.004267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:42.004267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:42.004337 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:42.004554 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:42.004555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:42.004556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:42.004679 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:42.004802 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:42.004896 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 15:15:42.004958 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":18}
I20260923 15:15:42.005971 11511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a1a5e3eb5574fcc8cc418ca695e5a3f: Generated new TSK 0
I20260923 15:15:42.006089 11511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:42.010857 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11431
W20260923 15:15:42.014787 11416 connection.cc:535] client connection to 127.21.204.126:41311 recv error: Network error: failed to read from TLS socket (remote: 127.21.204.126:41311): Connection reset by peer (error 104)
2026-09-23T15:15:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:42 dist-test-slave-t5lt krb5kdc[11530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:42 dist-test-slave-t5lt krb5kdc[11530](info): set up 2 sockets
Sep 23 15:15:42 dist-test-slave-t5lt krb5kdc[11530](info): commencing operation
krb5kdc: starting...
W20260923 15:15:43.043330 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:43 dist-test-slave-t5lt krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:43.161182 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:38843
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:38843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:43.241633 11543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:43.241858 11543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:43.241916 11543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:43.243044 11543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:43.243115 11543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:43.243142 11543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:43.243175 11543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:43.243198 11543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:43.244688 11543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:38843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:38843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:43.244958 11543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:43.245187 11543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:15:43.250181 11543 server_base.cc:1034] running on GCE node
W20260923 15:15:43.250296 11551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:43.250190 11549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:43.250197 11548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:43.250594 11543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:43.250823 11543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:43.251967 11543 hybrid_clock.cc:648] HybridClock initialized: now 1790176543251946 us; error 38 us; skew 500 ppm
Sep 23 15:15:43 dist-test-slave-t5lt krb5kdc[11530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176543, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:43.254290 11543 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:43.254773 11543 webserver.cc:460] Webserver started at http://127.21.204.126:45903/ using document root <none> and password file <none>
I20260923 15:15:43.255018 11543 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:43.255086 11543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:43.255237 11543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:43.256242 11543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "552f440b4145451c89c2260e5ea16188"
format_stamp: "Formatted at 2026-09-23 15:15:43 on dist-test-slave-t5lt"
server_key: "c02aa0109555ae46c34cc208ec8fb67c"
server_key_iv: "a4c09a0a7d1e1cfdb4d8102840f912aa"
server_key_version: "encryptionkey@0"
I20260923 15:15:43.256606 11543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "552f440b4145451c89c2260e5ea16188"
format_stamp: "Formatted at 2026-09-23 15:15:43 on dist-test-slave-t5lt"
server_key: "c02aa0109555ae46c34cc208ec8fb67c"
server_key_iv: "a4c09a0a7d1e1cfdb4d8102840f912aa"
server_key_version: "encryptionkey@0"
I20260923 15:15:43.257995 11543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:43.258803 11558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:43.259020 11543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:43.259110 11543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "552f440b4145451c89c2260e5ea16188"
format_stamp: "Formatted at 2026-09-23 15:15:43 on dist-test-slave-t5lt"
server_key: "c02aa0109555ae46c34cc208ec8fb67c"
server_key_iv: "a4c09a0a7d1e1cfdb4d8102840f912aa"
server_key_version: "encryptionkey@0"
I20260923 15:15:43.259162 11543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:43.267907 11543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:43.270744 11543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:43.270915 11543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:43.275285 11543 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:38843
I20260923 15:15:43.275316 11610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:38843 every 8 connection(s)
I20260923 15:15:43.275678 11543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:43.275967 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11543
I20260923 15:15:43.276079 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:43.276237 22321 external_mini_cluster.cc:1428] Setting key ea008a3abf7f846ce966e822c6a59c56
I20260923 15:15:43.277043 11611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:43.279858 11611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188: Bootstrap starting.
I20260923 15:15:43.280524 11611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:43.280858 11611 log.cc:824] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:43.281621 11611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188: No bootstrap required, opened a new log
I20260923 15:15:43.282049 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:43.276760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33291 (local address 127.21.204.126:38843)
0923 15:15:43.276960 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:43.276967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:43.276991 (+ 24us) server_negotiation.cc:407] Connection header received
0923 15:15:43.277463 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:43.277471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:43.277517 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:43.277627 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:43.277925 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:43.278816 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:43.279946 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:43.280413 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:43.280501 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:43.281240 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:43.281245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:43.281245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:43.281468 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:43.281498 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:43.281509 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:43.281580 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:43.281796 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 15:15:43.281828 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260923 15:15:43.282964 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:43.283675 11611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552f440b4145451c89c2260e5ea16188" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38843 } }
I20260923 15:15:43.283874 11611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:43.283936 11611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552f440b4145451c89c2260e5ea16188, State: Initialized, Role: FOLLOWER
I20260923 15:15:43.284061 11611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552f440b4145451c89c2260e5ea16188" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38843 } }
I20260923 15:15:43.284144 11611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:43.284188 11611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:43.284240 11611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:43.284916 11611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552f440b4145451c89c2260e5ea16188" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38843 } }
I20260923 15:15:43.285035 11611 leader_election.cc:302] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 552f440b4145451c89c2260e5ea16188; no voters:
I20260923 15:15:43.285220 11611 leader_election.cc:288] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:43.285259 11621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:43.285408 11621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [term 1 LEADER]: Becoming Leader. State: Replica: 552f440b4145451c89c2260e5ea16188, State: Running, Role: LEADER
I20260923 15:15:43.285459 11611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:43.285531 11621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552f440b4145451c89c2260e5ea16188" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38843 } }
I20260923 15:15:43.286015 11622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "552f440b4145451c89c2260e5ea16188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552f440b4145451c89c2260e5ea16188" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38843 } } }
I20260923 15:15:43.286026 11623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 552f440b4145451c89c2260e5ea16188. Latest consensus state: current_term: 1 leader_uuid: "552f440b4145451c89c2260e5ea16188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552f440b4145451c89c2260e5ea16188" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38843 } } }
I20260923 15:15:43.286151 11622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:43.286159 11623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:43.286350 11630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:43.287176 11630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:43.287639 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:43.284278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47203 (local address 127.21.204.126:38843)
0923 15:15:43.284318 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:43.284320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:43.284454 (+ 134us) server_negotiation.cc:407] Connection header received
0923 15:15:43.284488 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:43.284489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:43.284497 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:43.284540 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:43.284665 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:43.285093 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:43.286162 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:43.286456 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:43.286502 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:43.287276 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:43.287278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:43.287278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:43.287282 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:43.287296 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:43.287299 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:43.287331 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:43.287563 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 15:15:43.287574 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 15:15:43.289006 11630 catalog_manager.cc:1347] Generated new cluster ID: b42e9801e76542229377de63614cc5f1
I20260923 15:15:43.289060 11630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:43.321683 11630 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:43.322476 11630 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:43.327606 11630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 552f440b4145451c89c2260e5ea16188: Generated new TSK 0
I20260923 15:15:43.327785 11630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:43.332407 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11543
W20260923 15:15:43.337067 11528 connection.cc:535] client connection to 127.21.204.126:38843 recv error: Network error: failed to read from TLS socket (remote: 127.21.204.126:38843): Connection reset by peer (error 104)
2026-09-23T15:15:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:43 dist-test-slave-t5lt krb5kdc[11642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:43 dist-test-slave-t5lt krb5kdc[11642](info): set up 2 sockets
Sep 23 15:15:43 dist-test-slave-t5lt krb5kdc[11642](info): commencing operation
krb5kdc: starting...
W20260923 15:15:44.363242 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:44 dist-test-slave-t5lt krb5kdc[11642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:44.477248 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:45491
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:43233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:44.556035 11655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:44.556267 11655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:44.556308 11655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:44.557428 11655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:44.557492 11655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:44.557518 11655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:44.557544 11655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:44.557569 11655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:44.559074 11655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:43233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:45491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:45491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:44.559333 11655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:44.559545 11655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:44.564213 11660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:44.564219 11661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:44.564405 11655 server_base.cc:1034] running on GCE node
W20260923 15:15:44.564219 11663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:44.564664 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:44.564946 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:44.566108 11655 hybrid_clock.cc:648] HybridClock initialized: now 1790176544566103 us; error 36 us; skew 500 ppm
Sep 23 15:15:44 dist-test-slave-t5lt krb5kdc[11642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:44.568393 11655 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:44.568841 11655 webserver.cc:460] Webserver started at http://127.21.204.126:36447/ using document root <none> and password file <none>
I20260923 15:15:44.569073 11655 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:44.569139 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:44.569283 11655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:44.570333 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "d93f340b15064689aa133861b6b896f0"
format_stamp: "Formatted at 2026-09-23 15:15:44 on dist-test-slave-t5lt"
server_key: "ab13f243c5b55fe5a7521c12c4509771"
server_key_iv: "3e8c29ea336e5090552b59279f12c6c6"
server_key_version: "encryptionkey@0"
I20260923 15:15:44.570678 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "d93f340b15064689aa133861b6b896f0"
format_stamp: "Formatted at 2026-09-23 15:15:44 on dist-test-slave-t5lt"
server_key: "ab13f243c5b55fe5a7521c12c4509771"
server_key_iv: "3e8c29ea336e5090552b59279f12c6c6"
server_key_version: "encryptionkey@0"
I20260923 15:15:44.572074 11655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 15:15:44.572875 11670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:44.573063 11655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:44.573164 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "d93f340b15064689aa133861b6b896f0"
format_stamp: "Formatted at 2026-09-23 15:15:44 on dist-test-slave-t5lt"
server_key: "ab13f243c5b55fe5a7521c12c4509771"
server_key_iv: "3e8c29ea336e5090552b59279f12c6c6"
server_key_version: "encryptionkey@0"
I20260923 15:15:44.573236 11655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:44.576438 11655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:44.578197 11655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:44.578338 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:44.582182 11655 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:45491
I20260923 15:15:44.582211 11722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:45491 every 8 connection(s)
I20260923 15:15:44.582518 11655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:44.583423 11723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:44.585564 11723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0: Bootstrap starting.
I20260923 15:15:44.586195 11723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:44.586504 11723 log.cc:824] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:44.587177 11723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0: No bootstrap required, opened a new log
I20260923 15:15:44.588698 11723 raft_consensus.cc:348] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93f340b15064689aa133861b6b896f0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45491 } }
I20260923 15:15:44.588826 11723 raft_consensus.cc:374] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:44.588884 11723 raft_consensus.cc:729] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93f340b15064689aa133861b6b896f0, State: Initialized, Role: FOLLOWER
I20260923 15:15:44.589090 11723 consensus_queue.cc:260] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93f340b15064689aa133861b6b896f0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45491 } }
I20260923 15:15:44.589172 11723 raft_consensus.cc:388] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:44.589223 11723 raft_consensus.cc:482] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:44.589277 11723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:44.590173 11723 raft_consensus.cc:504] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93f340b15064689aa133861b6b896f0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45491 } }
I20260923 15:15:44.590286 11723 leader_election.cc:302] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d93f340b15064689aa133861b6b896f0; no voters:
I20260923 15:15:44.590466 11723 leader_election.cc:288] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:44.590494 11726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:44.590651 11726 raft_consensus.cc:686] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [term 1 LEADER]: Becoming Leader. State: Replica: d93f340b15064689aa133861b6b896f0, State: Running, Role: LEADER
I20260923 15:15:44.590720 11723 sys_catalog.cc:564] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:44.590780 11726 consensus_queue.cc:237] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93f340b15064689aa133861b6b896f0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45491 } }
I20260923 15:15:44.591123 11727 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d93f340b15064689aa133861b6b896f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93f340b15064689aa133861b6b896f0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45491 } } }
I20260923 15:15:44.591126 11728 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d93f340b15064689aa133861b6b896f0. Latest consensus state: current_term: 1 leader_uuid: "d93f340b15064689aa133861b6b896f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93f340b15064689aa133861b6b896f0" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45491 } } }
I20260923 15:15:44.591239 11727 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:44.591245 11728 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:44.591470 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11655
I20260923 15:15:44.591486 11733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:44.591583 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:44.591805 22321 external_mini_cluster.cc:1428] Setting key 8139d869ef9f75cf8d783638ee7abd5b
I20260923 15:15:44.592149 11733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:44.593758 11733 catalog_manager.cc:1347] Generated new cluster ID: 566e4be4dd5e4c9fb105c438dc32a769
I20260923 15:15:44.593801 11733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:44.594444 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:44.592428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49417 (local address 127.21.204.126:45491)
0923 15:15:44.592832 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:44.592842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:44.592877 (+ 35us) server_negotiation.cc:407] Connection header received
0923 15:15:44.593089 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:44.593098 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:44.593133 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:44.593285 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:44.594012 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:44.594015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:44.594016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:44.594020 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:44.594067 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:44.594078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:44.594148 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:44.594220 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 15:15:44.594262 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":68,"threads_started":1}
I20260923 15:15:44.595249 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:44.602994 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:44.596539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44655 (local address 127.21.204.126:45491)
0923 15:15:44.596573 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:44.596575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:44.602097 (+ 5522us) server_negotiation.cc:407] Connection header received
0923 15:15:44.602170 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:44.602170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:44.602176 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:44.602218 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:44.602789 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:44.602790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:44.602790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:44.602794 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:44.602808 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:44.602812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:44.602856 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:44.602908 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 15:15:44.602920 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:15:44.609134 11733 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:44.609313 11733 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:44.617969 11733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d93f340b15064689aa133861b6b896f0: Generated new TSK 0
I20260923 15:15:44.618122 11733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:44.624742 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11655
2026-09-23T15:15:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:44 dist-test-slave-t5lt krb5kdc[11754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:44 dist-test-slave-t5lt krb5kdc[11754](info): set up 2 sockets
Sep 23 15:15:44 dist-test-slave-t5lt krb5kdc[11754](info): commencing operation
krb5kdc: starting...
W20260923 15:15:45.655326 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:45 dist-test-slave-t5lt krb5kdc[11754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:45.770414 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35795
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:36903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:35795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:45.850374 11767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:45.850601 11767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:45.850642 11767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:45.851766 11767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:45.851831 11767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:45.851854 11767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:45.851881 11767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:45.851907 11767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:45.853360 11767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:36903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:35795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:45.853636 11767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:45.853864 11767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:45.858639 11772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:45.858665 11775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:45.858664 11773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:45.858709 11767 server_base.cc:1034] running on GCE node
I20260923 15:15:45.859073 11767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:45.859308 11767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:45.860458 11767 hybrid_clock.cc:648] HybridClock initialized: now 1790176545860443 us; error 42 us; skew 500 ppm
Sep 23 15:15:45 dist-test-slave-t5lt krb5kdc[11754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:45.863044 11767 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:45.863476 11767 webserver.cc:460] Webserver started at http://127.21.204.126:43229/ using document root <none> and password file <none>
I20260923 15:15:45.863718 11767 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:45.863790 11767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:45.863955 11767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:45.865008 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "c0da581548e34c14af807775758c6075"
format_stamp: "Formatted at 2026-09-23 15:15:45 on dist-test-slave-t5lt"
server_key: "405dc34329aa8086588169cbc37f4c4c"
server_key_iv: "f0282168d85e2bd8294cea38baf9a4a6"
server_key_version: "encryptionkey@0"
I20260923 15:15:45.865365 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "c0da581548e34c14af807775758c6075"
format_stamp: "Formatted at 2026-09-23 15:15:45 on dist-test-slave-t5lt"
server_key: "405dc34329aa8086588169cbc37f4c4c"
server_key_iv: "f0282168d85e2bd8294cea38baf9a4a6"
server_key_version: "encryptionkey@0"
I20260923 15:15:45.866820 11767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:45.867729 11782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:45.867962 11767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:15:45.868079 11767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "c0da581548e34c14af807775758c6075"
format_stamp: "Formatted at 2026-09-23 15:15:45 on dist-test-slave-t5lt"
server_key: "405dc34329aa8086588169cbc37f4c4c"
server_key_iv: "f0282168d85e2bd8294cea38baf9a4a6"
server_key_version: "encryptionkey@0"
I20260923 15:15:45.868156 11767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:45.881994 11767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:45.884064 11767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:45.884234 11767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:45.888160 11767 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:35795
I20260923 15:15:45.888191 11834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:35795 every 8 connection(s)
I20260923 15:15:45.888548 11767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:45.889423 11835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:45.891752 11835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075: Bootstrap starting.
I20260923 15:15:45.892433 11835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:45.892732 11835 log.cc:824] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:45.893435 11835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075: No bootstrap required, opened a new log
I20260923 15:15:45.894695 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11767
I20260923 15:15:45.894821 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:45.895007 22321 external_mini_cluster.cc:1428] Setting key 6a77e9690380aaac72ab43e1e9556666
I20260923 15:15:45.895586 11835 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0da581548e34c14af807775758c6075" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35795 } }
I20260923 15:15:45.895762 11835 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:45.895789 11835 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0da581548e34c14af807775758c6075, State: Initialized, Role: FOLLOWER
I20260923 15:15:45.895944 11835 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0da581548e34c14af807775758c6075" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35795 } }
I20260923 15:15:45.896015 11835 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:45.896036 11835 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:45.896099 11835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:45.896634 11835 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0da581548e34c14af807775758c6075" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35795 } }
I20260923 15:15:45.896754 11835 leader_election.cc:302] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0da581548e34c14af807775758c6075; no voters:
I20260923 15:15:45.896958 11835 leader_election.cc:288] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:45.896982 11840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:45.897095 11840 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [term 1 LEADER]: Becoming Leader. State: Replica: c0da581548e34c14af807775758c6075, State: Running, Role: LEADER
I20260923 15:15:45.897228 11835 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:45.897235 11840 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0da581548e34c14af807775758c6075" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35795 } }
I20260923 15:15:45.897680 11840 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0da581548e34c14af807775758c6075. Latest consensus state: current_term: 1 leader_uuid: "c0da581548e34c14af807775758c6075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0da581548e34c14af807775758c6075" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35795 } } }
I20260923 15:15:45.897686 11841 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0da581548e34c14af807775758c6075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0da581548e34c14af807775758c6075" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35795 } } }
I20260923 15:15:45.897851 11840 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:45.897861 11841 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:45.898236 11849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:45.898829 11849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:45.900552 11849 catalog_manager.cc:1347] Generated new cluster ID: 98d508b2a6fb4f2793023dd324bc5d1a
I20260923 15:15:45.900607 11849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:45 dist-test-slave-t5lt krb5kdc[11754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:45.906518 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:45.895519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34587 (local address 127.21.204.126:35795)
0923 15:15:45.895707 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:45.895715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:45.895740 (+ 25us) server_negotiation.cc:407] Connection header received
0923 15:15:45.895913 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:45.895920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:45.895975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:45.896139 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:45.896934 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:45.898327 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:45.899406 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:45.899886 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:45.900009 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:45.902802 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:45.902807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:45.902807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:45.902819 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:45.904784 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:45.905173 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:45.905176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:45.905176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:45.905296 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:45.905505 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:45.905506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:45.905507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:45.905811 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:45.906029 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:45.906109 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 15:15:45.906281 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260923 15:15:45.907469 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:45.917511 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:45.908965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44989 (local address 127.21.204.126:35795)
0923 15:15:45.909007 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:45.909008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:45.909110 (+ 102us) server_negotiation.cc:407] Connection header received
0923 15:15:45.909169 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:45.909170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:45.909179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:45.909230 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:45.909850 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:45.910627 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:45.911681 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:45.911991 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:45.912033 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:45.913800 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:45.913801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:45.913802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:45.913830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:45.916290 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:45.916670 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:45.916671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:45.916671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:45.916723 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:45.916920 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:45.916921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:45.916921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:45.917046 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:45.917127 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:45.917330 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 15:15:45.917410 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260923 15:15:45.929461 11849 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:45.930083 11849 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:45.943416 11849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0da581548e34c14af807775758c6075: Generated new TSK 0
I20260923 15:15:45.943621 11849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:45.948750 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11767
2026-09-23T15:15:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:46 dist-test-slave-t5lt krb5kdc[11866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:46 dist-test-slave-t5lt krb5kdc[11866](info): set up 2 sockets
Sep 23 15:15:46 dist-test-slave-t5lt krb5kdc[11866](info): commencing operation
krb5kdc: starting...
W20260923 15:15:46.991385 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:47 dist-test-slave-t5lt krb5kdc[11866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:47.108515 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:45027
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:47.188510 11879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:47.188727 11879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:47.188762 11879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:47.189908 11879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:47.189970 11879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:47.189993 11879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:47.190026 11879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:47.190073 11879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:47.191514 11879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:45027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:47.191773 11879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:47.191991 11879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:15:47.196828 11879 server_base.cc:1034] running on GCE node
W20260923 15:15:47.196863 11885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:47.196837 11884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:47.196843 11887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:47.197258 11879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:47.197548 11879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:47.198702 11879 hybrid_clock.cc:648] HybridClock initialized: now 1790176547198680 us; error 42 us; skew 500 ppm
Sep 23 15:15:47 dist-test-slave-t5lt krb5kdc[11866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:47.205637 11879 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:47.206127 11879 webserver.cc:460] Webserver started at http://127.21.204.126:42223/ using document root <none> and password file <none>
I20260923 15:15:47.206348 11879 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:47.206422 11879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:47.206569 11879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:47.207597 11879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "7d4cd7c70af74390bead08c779f5f255"
format_stamp: "Formatted at 2026-09-23 15:15:47 on dist-test-slave-t5lt"
server_key: "6aaf64c58abf5e69c108a553f6800edd"
server_key_iv: "0b878d644ad215d2826a0a5ca0db712e"
server_key_version: "encryptionkey@0"
I20260923 15:15:47.207949 11879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "7d4cd7c70af74390bead08c779f5f255"
format_stamp: "Formatted at 2026-09-23 15:15:47 on dist-test-slave-t5lt"
server_key: "6aaf64c58abf5e69c108a553f6800edd"
server_key_iv: "0b878d644ad215d2826a0a5ca0db712e"
server_key_version: "encryptionkey@0"
I20260923 15:15:47.209273 11879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:47.210045 11894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:47.210228 11879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:47.210335 11879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "7d4cd7c70af74390bead08c779f5f255"
format_stamp: "Formatted at 2026-09-23 15:15:47 on dist-test-slave-t5lt"
server_key: "6aaf64c58abf5e69c108a553f6800edd"
server_key_iv: "0b878d644ad215d2826a0a5ca0db712e"
server_key_version: "encryptionkey@0"
I20260923 15:15:47.210433 11879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:47.227334 11879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:47.229249 11879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:47.229415 11879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:47.233492 11879 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:45027
I20260923 15:15:47.233529 11946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:45027 every 8 connection(s)
I20260923 15:15:47.233927 11879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:47.234781 11947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:47.236943 11947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255: Bootstrap starting.
I20260923 15:15:47.237453 11947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:47.237692 11947 log.cc:824] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:47.238374 11947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255: No bootstrap required, opened a new log
I20260923 15:15:47.239915 11947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4cd7c70af74390bead08c779f5f255" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45027 } }
I20260923 15:15:47.240046 11947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:47.240067 11947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4cd7c70af74390bead08c779f5f255, State: Initialized, Role: FOLLOWER
I20260923 15:15:47.240164 11947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4cd7c70af74390bead08c779f5f255" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45027 } }
I20260923 15:15:47.240213 11947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:47.240234 11947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:47.240260 11947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:47.241048 11947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4cd7c70af74390bead08c779f5f255" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45027 } }
I20260923 15:15:47.241135 11947 leader_election.cc:302] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d4cd7c70af74390bead08c779f5f255; no voters:
I20260923 15:15:47.241258 11947 leader_election.cc:288] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:47.241325 11950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:47.241489 11950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4cd7c70af74390bead08c779f5f255, State: Running, Role: LEADER
I20260923 15:15:47.241530 11947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:47.241604 11950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4cd7c70af74390bead08c779f5f255" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45027 } }
I20260923 15:15:47.241954 11951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d4cd7c70af74390bead08c779f5f255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4cd7c70af74390bead08c779f5f255" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45027 } } }
I20260923 15:15:47.242053 11951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:47.241959 11952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d4cd7c70af74390bead08c779f5f255. Latest consensus state: current_term: 1 leader_uuid: "7d4cd7c70af74390bead08c779f5f255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4cd7c70af74390bead08c779f5f255" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 45027 } } }
I20260923 15:15:47.242141 11952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:47.242583 11957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:47.243074 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11879
I20260923 15:15:47.243101 11957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:47.243181 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:47.243417 22321 external_mini_cluster.cc:1428] Setting key 40854eefa0957443eb228f79dcaa24f7
I20260923 15:15:47.244695 11957 catalog_manager.cc:1347] Generated new cluster ID: ae12460ad6594a64a40da18e0dfdf1ce
I20260923 15:15:47.244748 11957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:47.249305 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:47.243846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40073 (local address 127.21.204.126:45027)
0923 15:15:47.244026 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:47.244037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:47.244068 (+ 31us) server_negotiation.cc:407] Connection header received
0923 15:15:47.244232 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:47.244240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:47.244322 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:47.244472 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:47.244727 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:47.246117 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:47.247231 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:47.247866 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:47.247985 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:47.248637 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:47.248641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:47.248642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:47.248845 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:47.248883 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:47.248898 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:47.248969 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:47.249035 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 15:15:47.249099 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260923 15:15:47.250242 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:47.255007 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:47.251628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50725 (local address 127.21.204.126:45027)
0923 15:15:47.251663 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:47.251666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:47.251708 (+ 42us) server_negotiation.cc:407] Connection header received
0923 15:15:47.251763 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:47.251764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:47.251773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:47.251835 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:47.251970 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:47.252691 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:47.253656 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:47.254004 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:47.254085 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:47.254711 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:47.254712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:47.254712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:47.254717 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:47.254730 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:47.254734 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:47.254788 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:47.254885 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 15:15:47.254906 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 15:15:47.255209 11957 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:47.255872 11957 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:47.262161 11957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d4cd7c70af74390bead08c779f5f255: Generated new TSK 0
I20260923 15:15:47.262343 11957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:47.266136 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11879
2026-09-23T15:15:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:47 dist-test-slave-t5lt krb5kdc[11978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:47 dist-test-slave-t5lt krb5kdc[11978](info): set up 2 sockets
Sep 23 15:15:47 dist-test-slave-t5lt krb5kdc[11978](info): commencing operation
krb5kdc: starting...
W20260923 15:15:48.303328 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:48 dist-test-slave-t5lt krb5kdc[11978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:48.420883 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35925
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:48.500906 11991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:48.501104 11991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:48.501127 11991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:48.502286 11991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:48.502326 11991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:48.502339 11991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:48.502351 11991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:48.502364 11991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:48.503890 11991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:48.504132 11991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:48.504347 11991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:48.509217 11997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:48.509262 11996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:48.509347 11991 server_base.cc:1034] running on GCE node
W20260923 15:15:48.509234 11999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:48.509694 11991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:48.510035 11991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:48.511230 11991 hybrid_clock.cc:648] HybridClock initialized: now 1790176548511230 us; error 40 us; skew 500 ppm
Sep 23 15:15:48 dist-test-slave-t5lt krb5kdc[11978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176548, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:48.513748 11991 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:48.514232 11991 webserver.cc:460] Webserver started at http://127.21.204.126:41317/ using document root <none> and password file <none>
I20260923 15:15:48.514479 11991 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:48.514549 11991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:48.514701 11991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:48.515770 11991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "46b937cd1a8143d5a712d0342d9d6eaa"
format_stamp: "Formatted at 2026-09-23 15:15:48 on dist-test-slave-t5lt"
server_key: "6afa6df4c144251b3de4cf45db10750e"
server_key_iv: "8463aab2176225eb7a8424eb74addd43"
server_key_version: "encryptionkey@0"
I20260923 15:15:48.516129 11991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "46b937cd1a8143d5a712d0342d9d6eaa"
format_stamp: "Formatted at 2026-09-23 15:15:48 on dist-test-slave-t5lt"
server_key: "6afa6df4c144251b3de4cf45db10750e"
server_key_iv: "8463aab2176225eb7a8424eb74addd43"
server_key_version: "encryptionkey@0"
I20260923 15:15:48.517493 11991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:48.518290 12006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:48.518493 11991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 15:15:48.518579 11991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "46b937cd1a8143d5a712d0342d9d6eaa"
format_stamp: "Formatted at 2026-09-23 15:15:48 on dist-test-slave-t5lt"
server_key: "6afa6df4c144251b3de4cf45db10750e"
server_key_iv: "8463aab2176225eb7a8424eb74addd43"
server_key_version: "encryptionkey@0"
I20260923 15:15:48.518625 11991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:48.521783 11991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:48.523685 11991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:48.523789 11991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:48.527527 11991 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:35925
I20260923 15:15:48.527618 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:35925 every 8 connection(s)
I20260923 15:15:48.527913 11991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:48.528926 12059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:48.531172 12059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa: Bootstrap starting.
I20260923 15:15:48.531752 12059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:48.532044 12059 log.cc:824] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:48.532770 12059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa: No bootstrap required, opened a new log
I20260923 15:15:48.534403 12059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35925 } }
I20260923 15:15:48.534557 12059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 15:15:48.534601 12059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b937cd1a8143d5a712d0342d9d6eaa, State: Initialized, Role: FOLLOWER
I20260923 15:15:48.534767 12059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35925 } }
I20260923 15:15:48.534861 12059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 15:15:48.534906 12059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 15:15:48.534960 12059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:48.535195 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 11991
I20260923 15:15:48.535319 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:48.535540 22321 external_mini_cluster.cc:1428] Setting key 40d047deeb6e0f3117cee56ff13a5f24
I20260923 15:15:48.535950 12059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35925 } }
I20260923 15:15:48.536095 12059 leader_election.cc:302] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b937cd1a8143d5a712d0342d9d6eaa; no voters:
I20260923 15:15:48.536257 12059 leader_election.cc:288] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 15:15:48.536306 12064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:48.536458 12064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [term 1 LEADER]: Becoming Leader. State: Replica: 46b937cd1a8143d5a712d0342d9d6eaa, State: Running, Role: LEADER
I20260923 15:15:48.536496 12059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:48.536579 12064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35925 } }
I20260923 15:15:48.537014 12065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35925 } } }
I20260923 15:15:48.537088 12065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:48.537518 12068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:48.537282 12066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b937cd1a8143d5a712d0342d9d6eaa. Latest consensus state: current_term: 1 leader_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b937cd1a8143d5a712d0342d9d6eaa" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 35925 } } }
I20260923 15:15:48.537659 12066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:48.537962 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:48.535984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35793 (local address 127.21.204.126:35925)
0923 15:15:48.536159 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:48.536168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:48.536202 (+ 34us) server_negotiation.cc:407] Connection header received
0923 15:15:48.536319 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:48.536327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:48.536372 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:48.536825 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:48.537268 (+ 443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:48.537271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:48.537272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:48.537276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:48.537321 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:48.537337 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:48.537663 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:48.537697 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 15:15:48.537735 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260923 15:15:48.538054 12068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:48.539040 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:48.539801 12068 catalog_manager.cc:1347] Generated new cluster ID: 346006099af74e279897b8c35ed648be
I20260923 15:15:48.539855 12068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:48.541616 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:48.540548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43075 (local address 127.21.204.126:35925)
0923 15:15:48.540582 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:48.540584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:48.540696 (+ 112us) server_negotiation.cc:407] Connection header received
0923 15:15:48.540764 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:48.540765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:48.540770 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:48.540823 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:48.541402 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:48.541404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:48.541404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 15:15:48.541408 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:48.541420 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 15:15:48.541424 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:48.541473 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:48.541525 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 15:15:48.541539 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:15:48.553507 12068 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:48.553656 12068 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:48.569772 12068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46b937cd1a8143d5a712d0342d9d6eaa: Generated new TSK 0
I20260923 15:15:48.569995 12068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 15:15:48.572297 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 11991
2026-09-23T15:15:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1306 ms)
[----------] 18 tests from AuthTokenIssuingTest (23668 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:48 dist-test-slave-t5lt krb5kdc[12090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:48 dist-test-slave-t5lt krb5kdc[12090](info): set up 2 sockets
Sep 23 15:15:48 dist-test-slave-t5lt krb5kdc[12090](info): commencing operation
krb5kdc: starting...
W20260923 15:15:49.603361 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:49 dist-test-slave-t5lt krb5kdc[12090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:49.718077 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33087
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:33087,127.21.204.125:33263,127.21.204.124:45463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:49.798616 12103 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 15:15:49.798806 12103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:49.798823 12103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:49.798843 12103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:49.799993 12103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:49.800033 12103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:49.800048 12103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:49.800089 12103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:49.800117 12103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:49.801527 12103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:33087,127.21.204.125:33263,127.21.204.124:45463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:49.801765 12103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:49.801998 12103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:49.806670 12109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:49.806663 12111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:49.806663 12108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:49.806838 12103 server_base.cc:1034] running on GCE node
I20260923 15:15:49.807060 12103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:49.807356 12103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:49.808542 12103 hybrid_clock.cc:648] HybridClock initialized: now 1790176549808527 us; error 44 us; skew 500 ppm
Sep 23 15:15:49 dist-test-slave-t5lt krb5kdc[12090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:49.810946 12103 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:49.811385 12103 webserver.cc:460] Webserver started at http://127.21.204.126:40713/ using document root <none> and password file <none>
I20260923 15:15:49.811602 12103 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:49.811666 12103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:49.811808 12103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:49.812801 12103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9"
format_stamp: "Formatted at 2026-09-23 15:15:49 on dist-test-slave-t5lt"
server_key: "b6a8e21962b8a38f596a732994e7ec88"
server_key_iv: "1e190a85c25fd4a1beea9cc3de682fc6"
server_key_version: "encryptionkey@0"
I20260923 15:15:49.813158 12103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9"
format_stamp: "Formatted at 2026-09-23 15:15:49 on dist-test-slave-t5lt"
server_key: "b6a8e21962b8a38f596a732994e7ec88"
server_key_iv: "1e190a85c25fd4a1beea9cc3de682fc6"
server_key_version: "encryptionkey@0"
I20260923 15:15:49.814512 12103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:49.815240 12118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:49.815408 12103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 15:15:49.815512 12103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9"
format_stamp: "Formatted at 2026-09-23 15:15:49 on dist-test-slave-t5lt"
server_key: "b6a8e21962b8a38f596a732994e7ec88"
server_key_iv: "1e190a85c25fd4a1beea9cc3de682fc6"
server_key_version: "encryptionkey@0"
I20260923 15:15:49.815585 12103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:49.822811 12103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:49.824525 12103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:49.824681 12103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:49.828848 12103 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:33087
I20260923 15:15:49.828893 12170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:33087 every 8 connection(s)
I20260923 15:15:49.829232 12103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:49.830180 12171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:49.831648 12171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:15:49.832221 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 12103
I20260923 15:15:49.832356 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:49.832567 22321 external_mini_cluster.cc:1428] Setting key 9c82c833489289a573405903becdc6a2
Sep 23 15:15:49 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.125@KRBTEST.COM; defaulting to no policy
I20260923 15:15:49.842052 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.832387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33087 (local address 127.0.0.1:39936)
0923 15:15:49.832601 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.832645 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:49.833162 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:49.833468 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:49.833470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:49.833708 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:49.833947 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:49.833953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.835028 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.835029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:49.835712 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:49.835715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.836440 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.836441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:49.836506 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:49.837045 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:49.837053 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:49.838639 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:49.840678 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:49.840685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:49.840687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:49.840885 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:49.841101 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:49.841101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:49.841101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:49.841176 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:49.841505 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:49.841510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:49.841598 (+ 88us) client_negotiation.cc:769] Sending connection context
0923 15:15:49.841697 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 15:15:49.841816 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260923 15:15:49.842068 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.832616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.21.204.126:33087)
0923 15:15:49.832760 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:49.832765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:49.832808 (+ 43us) server_negotiation.cc:407] Connection header received
0923 15:15:49.833320 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:49.833323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:49.833345 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:49.833449 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:49.834083 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.834932 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.835803 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.836333 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.836448 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:49.838730 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:49.838739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:49.838739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:49.838748 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:49.840603 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:49.840963 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:49.840966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:49.840966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:49.841041 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:49.841233 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:49.841234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:49.841234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:49.841421 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:49.841510 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:49.841739 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 15:15:49.841924 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.125@KRBTEST.COM" created.
I20260923 15:15:49.842972 12171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:15:49.843276 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.843173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42920)
0923 15:15:49.843216 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.843257 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 15:15:49.843425 12121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
I20260923 15:15:49.843823 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.843742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42924)
0923 15:15:49.843784 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.843807 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 15:15:49.843947 12171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:33263: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab.
Entry for principal kudu/127.21.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab.
I20260923 15:15:49.857214 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:33263
--webserver_interface=127.21.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab
--principal=kudu/127.21.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:33087,127.21.204.125:33263,127.21.204.124:45463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:15:49.893216 12171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } attempt: 1
I20260923 15:15:49.893772 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.893618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42934)
0923 15:15:49.893684 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.893733 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 15:15:49.894173 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.894071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42938)
0923 15:15:49.894121 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.894149 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 15:15:49.894272 12171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:33263: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
W20260923 15:15:49.938864 12178 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 15:15:49.939059 12178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:49.939076 12178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:49.939096 12178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:49.940148 12178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:49.940181 12178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:49.940196 12178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:49.940208 12178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:49.940220 12178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:49.941618 12178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:33087,127.21.204.125:33263,127.21.204.124:45463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab
--principal=kudu/127.21.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:33263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:49.941795 12178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:49.941967 12178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:49.946800 12186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:49.946805 12184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:49.946856 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:49.947017 12178 server_base.cc:1034] running on GCE node
I20260923 15:15:49.947255 12178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:49.947528 12178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:49.948712 12178 hybrid_clock.cc:648] HybridClock initialized: now 1790176549948683 us; error 40 us; skew 500 ppm
Sep 23 15:15:49 dist-test-slave-t5lt krb5kdc[12090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:49.951197 12178 init.cc:377] Logged in from keytab as kudu/127.21.204.125@KRBTEST.COM (short username kudu)
I20260923 15:15:49.951628 12178 webserver.cc:460] Webserver started at http://127.21.204.125:46697/ using document root <none> and password file <none>
I20260923 15:15:49.951843 12178 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:49.951889 12178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:49.952047 12178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:49.953066 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data/instance:
uuid: "859680366f7c4305b6993ae93a9485b6"
format_stamp: "Formatted at 2026-09-23 15:15:49 on dist-test-slave-t5lt"
server_key: "b8feb10aa32ddce18178286d3da5ff3d"
server_key_iv: "90d635bc4f53e12b6cab64778615bbbb"
server_key_version: "encryptionkey@0"
I20260923 15:15:49.953418 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/wal/instance:
uuid: "859680366f7c4305b6993ae93a9485b6"
format_stamp: "Formatted at 2026-09-23 15:15:49 on dist-test-slave-t5lt"
server_key: "b8feb10aa32ddce18178286d3da5ff3d"
server_key_iv: "90d635bc4f53e12b6cab64778615bbbb"
server_key_version: "encryptionkey@0"
I20260923 15:15:49.954794 12178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:49.955483 12193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:49.955674 12178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:49.955757 12178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/wal
uuid: "859680366f7c4305b6993ae93a9485b6"
format_stamp: "Formatted at 2026-09-23 15:15:49 on dist-test-slave-t5lt"
server_key: "b8feb10aa32ddce18178286d3da5ff3d"
server_key_iv: "90d635bc4f53e12b6cab64778615bbbb"
server_key_version: "encryptionkey@0"
I20260923 15:15:49.955811 12178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:49.962402 12171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } attempt: 2
I20260923 15:15:49.962750 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.962628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42954)
0923 15:15:49.962685 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.962721 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 15:15:49.963086 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.963006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42962)
0923 15:15:49.963046 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.963069 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 15:15:49.963210 12171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:33263: Network error: Client connection negotiation failed: client connection to 127.21.204.125:33263: connect: Connection refused (error 111)
I20260923 15:15:49.967300 12178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:49.969182 12178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:49.969333 12178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:49.973251 12178 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.125:33263
I20260923 15:15:49.973291 12245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.125:33263 every 8 connection(s)
I20260923 15:15:49.973649 12178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
I20260923 15:15:49.974550 12246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:49.976011 12246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:49 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:49.981381 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 12178
I20260923 15:15:49.981462 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-1/wal/instance
I20260923 15:15:49.981652 22321 external_mini_cluster.cc:1428] Setting key 92d49b208907f6cbab520247178fd517
I20260923 15:15:49.984846 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.976684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.21.204.126:33087)
0923 15:15:49.976737 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:49.976739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:49.976916 (+ 177us) server_negotiation.cc:407] Connection header received
0923 15:15:49.977358 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:49.977359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:49.977370 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:49.977431 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:49.978063 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.978533 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.979428 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.979622 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.979674 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:49.981701 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:49.981702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:49.981702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:49.981709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:49.983427 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:49.983808 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:49.983810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:49.983810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:49.983878 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:49.984185 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:49.984186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:49.984186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:49.984292 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:49.984344 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:49.984686 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 15:15:49.984749 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 15:15:49.985100 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.976637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33087 (local address 127.0.0.1:39950)
0923 15:15:49.976790 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.976831 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:49.977281 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:49.977504 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:49.977507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:49.977729 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:49.977989 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:49.977994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.978605 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.978606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:49.979352 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:49.979354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.979692 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.979692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:49.979784 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:49.980203 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:49.980211 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:49.981595 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:49.983514 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:49.983520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:49.983522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:49.983731 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:49.983976 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:49.983977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:49.983978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:49.984118 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:49.984359 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:49.984365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:49.984552 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 15:15:49.984667 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 15:15:49.984837 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260923 15:15:49.985610 12246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.204.124@KRBTEST.COM; defaulting to no policy
Sep 23 15:15:49 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.124@KRBTEST.COM" created.
I20260923 15:15:49.993770 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.985943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.21.204.125:33263)
0923 15:15:49.986092 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:49.986097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:49.986115 (+ 18us) server_negotiation.cc:407] Connection header received
0923 15:15:49.986142 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:49.986144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:49.986154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:49.986216 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:49.986567 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.987230 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.988220 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.988704 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.988773 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:49.990810 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:49.990817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:49.990817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:49.990823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:49.992376 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:49.992732 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:49.992735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:49.992736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:49.992819 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:49.993063 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:49.993064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:49.993064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:49.993251 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:49.993326 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:49.993510 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 15:15:49.993643 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260923 15:15:49.993856 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.985868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42972)
0923 15:15:49.985943 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.985955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:49.986001 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:49.986245 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:49.986246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:49.986405 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:49.986479 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:49.986481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.987311 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.987312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:49.988134 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:49.988137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:49.988809 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:49.988810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:49.988936 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:49.989317 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:49.989322 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:49.990713 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:49.992486 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:49.992487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:49.992487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:49.992647 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:49.992890 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:49.992891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:49.992891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:49.992978 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:49.993336 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:49.993337 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:49.993386 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 15:15:49.993484 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 15:15:49.993594 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 15:15:49.994727 12246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:15:49.995154 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.995027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53410)
0923 15:15:49.995080 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.995126 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 15:15:49.995287 12197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
I20260923 15:15:49.995735 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:49.995635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53426)
0923 15:15:49.995684 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:49.995717 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 15:15:49.995837 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:45463: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab.
Entry for principal kudu/127.21.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab.
I20260923 15:15:50.008708 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:45463
--webserver_interface=127.21.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab
--principal=kudu/127.21.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:44199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:33087,127.21.204.125:33263,127.21.204.124:45463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:15:50.045076 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } attempt: 1
I20260923 15:15:50.045647 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.045499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53432)
0923 15:15:50.045573 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.045615 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 15:15:50.046051 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.045950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53444)
0923 15:15:50.045995 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.046028 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 15:15:50.046140 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:45463: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
I20260923 15:15:50.054373 12171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } attempt: 3
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
I20260923 15:15:50.061537 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.054768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.21.204.125:33263)
0923 15:15:50.054832 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.054835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.054853 (+ 18us) server_negotiation.cc:407] Connection header received
0923 15:15:50.054902 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.054903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.054921 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.054975 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.055308 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.055810 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.056545 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.056773 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.056852 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.058821 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.058822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.058823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.058832 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.060388 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.060756 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.060757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.060758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.060819 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.061020 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.061021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.061021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.061136 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.061212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.061367 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.061444 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 15:15:50.061540 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.054700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42986)
0923 15:15:50.054752 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.054769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:50.054837 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:50.054996 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:50.054997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:50.055157 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:50.055225 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.055227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.055910 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.055911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.056462 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.056464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.056858 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.056859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.056940 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.057439 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:50.057443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:50.058730 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:50.060501 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.060502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.060503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.060678 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.060890 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.060891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.060891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.060954 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.061205 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:50.061206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:50.061254 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 15:15:50.061341 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 15:15:50.061444 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 15:15:50.062034 12171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:15:50.062381 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.062279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53458)
0923 15:15:50.062320 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.062355 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 15:15:50.062692 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.062636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53472)
0923 15:15:50.062659 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.062679 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260923 15:15:50.062793 12171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:45463: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
W20260923 15:15:50.090022 12253 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 15:15:50.090231 12253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:50.090248 12253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:50.090268 12253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:50.091500 12253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:50.091542 12253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:50.091555 12253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:50.091567 12253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:50.091579 12253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:50.093032 12253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:44199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:33087,127.21.204.125:33263,127.21.204.124:45463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab
--principal=kudu/127.21.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:50.093271 12253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:50.093508 12253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 15:15:50.093952 12171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } attempt: 1
I20260923 15:15:50.094384 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.094254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53486)
0923 15:15:50.094316 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.094355 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 15:15:50.094805 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.094690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53498)
0923 15:15:50.094748 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.094785 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 15:15:50.094934 12171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:45463: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
W20260923 15:15:50.098312 12259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:50.098373 12258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:50.098306 12261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:50.098717 12253 server_base.cc:1034] running on GCE node
I20260923 15:15:50.098941 12253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:50.099161 12253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:50.100333 12253 hybrid_clock.cc:648] HybridClock initialized: now 1790176550100321 us; error 30 us; skew 500 ppm
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:50.102921 12253 init.cc:377] Logged in from keytab as kudu/127.21.204.124@KRBTEST.COM (short username kudu)
I20260923 15:15:50.103339 12253 webserver.cc:460] Webserver started at http://127.21.204.124:44669/ using document root <none> and password file <none>
I20260923 15:15:50.103569 12253 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:50.103614 12253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:50.103758 12253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:50.104791 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data/instance:
uuid: "29a7f757c977405db7245bf123716320"
format_stamp: "Formatted at 2026-09-23 15:15:50 on dist-test-slave-t5lt"
server_key: "ced9f02eb3eca5f033c1a36ffc6e3380"
server_key_iv: "35d0a8245576c47b28e1c3bf5139f021"
server_key_version: "encryptionkey@0"
I20260923 15:15:50.105158 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/wal/instance:
uuid: "29a7f757c977405db7245bf123716320"
format_stamp: "Formatted at 2026-09-23 15:15:50 on dist-test-slave-t5lt"
server_key: "ced9f02eb3eca5f033c1a36ffc6e3380"
server_key_iv: "35d0a8245576c47b28e1c3bf5139f021"
server_key_version: "encryptionkey@0"
I20260923 15:15:50.106600 12253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:50.107381 12268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:50.107568 12253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:50.107674 12253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/wal
uuid: "29a7f757c977405db7245bf123716320"
format_stamp: "Formatted at 2026-09-23 15:15:50 on dist-test-slave-t5lt"
server_key: "ced9f02eb3eca5f033c1a36ffc6e3380"
server_key_iv: "35d0a8245576c47b28e1c3bf5139f021"
server_key_version: "encryptionkey@0"
I20260923 15:15:50.107750 12253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:50.114275 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } attempt: 2
I20260923 15:15:50.114655 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.114534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53514)
0923 15:15:50.114592 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.114627 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 15:15:50.114988 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.114888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53524)
0923 15:15:50.114938 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.114962 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 15:15:50.115068 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:45463: Network error: Client connection negotiation failed: client connection to 127.21.204.124:45463: connect: Connection refused (error 111)
I20260923 15:15:50.128207 12253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:50.130532 12253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:50.130694 12253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:50.134684 12253 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.124:45463
I20260923 15:15:50.134716 12320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.124:45463 every 8 connection(s)
I20260923 15:15:50.135063 12253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
I20260923 15:15:50.135946 12321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:50.137514 12321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:50.143378 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 12253
I20260923 15:15:50.143493 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790176278407460-22321-0/minicluster-data/master-2/wal/instance
I20260923 15:15:50.143676 22321 external_mini_cluster.cc:1428] Setting key e4f3da0499c68fda19eb8945d64419aa
I20260923 15:15:50.146901 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.138295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.21.204.126:33087)
0923 15:15:50.138379 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.138381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.138639 (+ 258us) server_negotiation.cc:407] Connection header received
0923 15:15:50.139107 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.139108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.139120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.139179 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.139778 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.140251 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.141235 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.141443 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.141501 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.143774 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.143776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.143776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.143784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.145491 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.145957 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.145958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.145958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.146011 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.146268 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.146268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.146268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.146371 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.146422 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.146726 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.146808 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 15:15:50.146937 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.138258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:33087 (local address 127.0.0.1:39960)
0923 15:15:50.138488 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.138530 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:50.138992 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:50.139246 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:50.139250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:50.139487 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:50.139682 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.139686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.140376 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.140377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.141155 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.141157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.141522 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.141522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.141618 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.142190 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:50.142199 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:50.143673 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:50.145565 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.145570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.145573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.145855 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.146074 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.146075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.146076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.146196 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.146440 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:50.146445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:50.146599 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 15:15:50.146677 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 15:15:50.146787 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260923 15:15:50.147526 12321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
I20260923 15:15:50.153252 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.144090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.21.204.124:45463)
0923 15:15:50.144235 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.144239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.144413 (+ 174us) server_negotiation.cc:407] Connection header received
0923 15:15:50.144501 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.144505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.144530 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.144594 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.145395 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.146455 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.147631 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.148066 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.148147 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.150257 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.150265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.150265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.150273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.152101 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.152438 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.152442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.152442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.152507 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.152708 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.152709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.152709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.152847 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.153015 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.153064 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.153141 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260923 15:15:50.154335 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.147788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:33263 (local address 127.0.0.1:42990)
0923 15:15:50.147839 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.147851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:50.147891 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:50.148042 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:50.148042 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:50.148169 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:50.148216 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.148217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.148979 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.148980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.149453 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.149455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.149829 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.149830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.149916 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.150337 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:50.150341 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:50.151456 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:50.153091 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.153092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.153093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.153307 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.153627 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.153628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.153628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.153705 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.154046 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:50.154047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:50.154092 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 15:15:50.154161 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 15:15:50.154249 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 15:15:50.154364 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.147888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.21.204.125:33263)
0923 15:15:50.147935 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.147938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.147945 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:15:50.147967 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.147968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.147977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.148030 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.148300 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.148883 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.149539 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.149742 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.149828 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.151545 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.151546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.151546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.151554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.152990 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.153483 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.153485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.153485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.153556 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.153807 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.153808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.153808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.153974 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.154040 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.154202 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.154274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:50.154752 12321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:15:50.162629 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.155021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.21.204.124:45463)
0923 15:15:50.155078 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.155079 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.155108 (+ 29us) server_negotiation.cc:407] Connection header received
0923 15:15:50.155173 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.155174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.155181 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.155221 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.155439 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.156103 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.156919 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.157247 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.157381 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.159719 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.159720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.159721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.159728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.161296 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.161696 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.161697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.161697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.161752 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.161994 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.161995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.161996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.162138 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.162219 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.162434 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.162540 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":41}
I20260923 15:15:50.162643 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.154085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.21.204.126:33087)
0923 15:15:50.154128 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.154130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.154138 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:15:50.154158 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.154159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.154167 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.154214 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.154798 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.155529 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.156434 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.156722 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.156793 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.159156 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.159158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.159158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.159165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.161022 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.161345 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.161346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.161346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.161412 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.161614 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.161615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.161615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.161755 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.162422 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.162469 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.162547 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 15:15:50.162633 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.154964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53546)
0923 15:15:50.155055 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.155064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:50.155122 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:50.155225 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:50.155225 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:50.155356 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:50.155385 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.155386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.156176 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.156176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.156835 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.156837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.157375 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.157376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.157415 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.157816 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:50.157821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:50.159614 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:50.161401 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.161402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.161403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.161625 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.161829 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.161829 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.161830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.161921 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.162250 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:50.162251 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:50.162295 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 15:15:50.162396 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 15:15:50.162499 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 15:15:50.164476 12321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: Bootstrap starting.
I20260923 15:15:50.165436 12321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:50.165722 12321 log.cc:824] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:50.166600 12321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: No bootstrap required, opened a new log
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
I20260923 15:15:50.168102 12321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.168292 12321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a7f757c977405db7245bf123716320, State: Initialized, Role: FOLLOWER
I20260923 15:15:50.168375 12321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.168699 12321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:50.168653 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.168826 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:50.170053 12171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } attempt: 2
I20260923 15:15:50.171123 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.163347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.21.204.125:33263)
0923 15:15:50.163384 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.163385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.163417 (+ 32us) server_negotiation.cc:407] Connection header received
0923 15:15:50.163444 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.163445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.163454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.163495 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.164041 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.164493 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.165213 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.165452 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.165546 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.167924 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.167925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.167925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.167933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.169575 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.170007 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.170009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.170009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.170073 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.170292 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.170293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.170293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.170427 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.170591 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.170961 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.171032 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 15:15:50.172035 12338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:15:50.172137 12338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:50.172344 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:50.171104 12253 master_runner.cc:186] Error getting master registration for 127.21.204.126:33087: OK, instance_id { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" instance_seqno: 1790176549827507 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 15:15:50.173844 12253 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 15:15:50.173267 12145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39966
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:15:50.178452 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.170429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.21.204.124:45463)
0923 15:15:50.170465 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.170467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.170474 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:15:50.170493 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.170494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.170502 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.170548 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.170872 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.171525 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.172432 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.172746 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.172832 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.175843 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.175845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.175845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.175851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.177259 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.177617 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.177618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.177618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.177675 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.177930 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.177931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.177931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.178056 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.178137 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.178297 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.178370 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 15:15:50.178470 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.170321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53558)
0923 15:15:50.170357 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.170368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:50.170418 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:50.170595 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:50.170596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:50.170761 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:50.170800 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.170802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.171616 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.171616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.172347 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.172350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.172840 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.172841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.172905 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.174318 (+ 1413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:50.174322 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:50.175735 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:50.177363 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.177363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.177364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.177541 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.177741 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.177742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.177743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.177800 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.178143 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:50.178144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:50.178187 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 15:15:50.178272 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:15:50.178369 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260923 15:15:50.180194 12171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9: Bootstrap starting.
I20260923 15:15:50.180871 12171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:50.181157 12171 log.cc:824] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:50.181993 12171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9: No bootstrap required, opened a new log
I20260923 15:15:50.183336 12171 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.183466 12171 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f1b3ed6ec14a39bd3b2f8751e569c9, State: Initialized, Role: FOLLOWER
I20260923 15:15:50.183540 12171 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.183830 12171 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:50.183871 12145 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 15:15:50.183941 12145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:50.184796 12145 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.185067 12145 leader_election.cc:288] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [CANDIDATE]: Term 1 election: Requested vote from peers 859680366f7c4305b6993ae93a9485b6 (127.21.204.125:33263), 29a7f757c977405db7245bf123716320 (127.21.204.124:45463)
W20260923 15:15:50.185869 12352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:15:50.185959 12352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:50.183786 12341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.186040 12341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:50.186800 12295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29a7f757c977405db7245bf123716320"
I20260923 15:15:50.186934 12295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:50.187765 12295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f1b3ed6ec14a39bd3b2f8751e569c9 in term 1.
I20260923 15:15:50.188031 12122 leader_election.cc:302] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7f757c977405db7245bf123716320, e2f1b3ed6ec14a39bd3b2f8751e569c9; no voters:
I20260923 15:15:50.188140 12341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:50.188261 12341 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [term 1 LEADER]: Becoming Leader. State: Replica: e2f1b3ed6ec14a39bd3b2f8751e569c9, State: Running, Role: LEADER
I20260923 15:15:50.188333 12341 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.188709 12358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2f1b3ed6ec14a39bd3b2f8751e569c9. Latest consensus state: current_term: 1 leader_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.188884 12358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:50.186733 12103 master_runner.cc:186] Error getting master registration for 127.21.204.125:33263: OK, instance_id { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" instance_seqno: 1790176549972211 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 15:15:50.189016 12103 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 15:15:50.201730 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.190521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.21.204.125:33263)
0923 15:15:50.190577 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.190579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.194351 (+ 3772us) server_negotiation.cc:407] Connection header received
0923 15:15:50.194441 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.194442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.194452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.194481 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.195134 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.195877 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.196949 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.197260 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.197300 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.198875 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.198877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.198877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.198884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.200526 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.200919 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.200920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.200921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.200974 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.201172 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.201173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.201173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.201292 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.201359 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.201561 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.201636 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:15:50.202522 12360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:50.202981 12360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:50.203341 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.190153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.21.204.124:45463)
0923 15:15:50.190208 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.190210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.194075 (+ 3865us) server_negotiation.cc:407] Connection header received
0923 15:15:50.194155 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.194156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.194181 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.194234 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.195410 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.196112 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.197738 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.198057 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.198100 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.200172 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.200186 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.200187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.200194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.201987 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.202533 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.202534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.202535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.202591 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.202784 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.202785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.202785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.202899 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.202974 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.203161 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.203227 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:50.205138 12295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [term 1 FOLLOWER]: Refusing update from remote peer e2f1b3ed6ec14a39bd3b2f8751e569c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:50.205468 12358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:50.206195 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } attempt: 3
I20260923 15:15:50.207275 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2f1b3ed6ec14a39bd3b2f8751e569c9. Latest consensus state: current_term: 1 leader_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.207358 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:50.208276 12341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.208385 12341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:50.208503 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.208596 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:50.208775 12365 mvcc.cc:205] Tried to move back new op lower bound from 7332563149637414912 to 7332563149572022272. Current Snapshot: MvccSnapshot[applied={T|T < 7332563149637414912}]
I20260923 15:15:50.209085 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.190366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.21.204.126:33087)
0923 15:15:50.190403 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.190405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.194216 (+ 3811us) server_negotiation.cc:407] Connection header received
0923 15:15:50.194277 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.194278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.194289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.194340 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.202534 (+ 8194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.203242 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.204168 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.204467 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.204540 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.206011 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.206013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.206013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.206023 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.207629 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.208015 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.208015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.208016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.208075 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.208280 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.208280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.208281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.208431 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.208531 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.208907 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.208990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 23 15:15:50 dist-test-slave-t5lt krb5kdc[12090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176549, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:15:50.213907 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.206420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:45463 (local address 127.0.0.1:53580)
0923 15:15:50.206500 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:50.206510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:50.206547 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:50.206791 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:50.206792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:50.206949 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:50.206992 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.206994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.207807 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.207808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.208543 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:50.208545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.209015 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.209016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:50.209067 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.209609 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:50.209614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:50.211098 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:50.212907 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.212909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.212909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.213118 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.213305 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:50.213305 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:50.213305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:50.213355 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:50.213603 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:50.213603 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:50.213643 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 15:15:50.213701 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 15:15:50.213787 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 15:15:50.214238 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:50.206631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.21.204.124:45463)
0923 15:15:50.206672 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.206674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.206681 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:15:50.206697 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.206698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.206707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.206753 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.207075 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.207715 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.208632 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:50.208901 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:50.208964 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:50.211239 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:50.211240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:50.211241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:50.211247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:50.212818 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.213193 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.213194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.213194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.213243 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:50.213414 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:50.213415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:50.213415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:50.213525 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:50.213579 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:50.214089 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 15:15:50.214157 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 15:15:50.215596 12246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: Bootstrap starting.
I20260923 15:15:50.216277 12246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:50.216554 12246 log.cc:824] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:50.217499 12246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: No bootstrap required, opened a new log
I20260923 15:15:50.219578 12246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.219719 12246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859680366f7c4305b6993ae93a9485b6, State: Initialized, Role: FOLLOWER
I20260923 15:15:50.219799 12246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } }
I20260923 15:15:50.220105 12246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:50.220204 12219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:50.221033 12219 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [term 1 FOLLOWER]: Refusing update from remote peer e2f1b3ed6ec14a39bd3b2f8751e569c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:50.221196 12220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "859680366f7c4305b6993ae93a9485b6"
I20260923 15:15:50.222244 12220 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f1b3ed6ec14a39bd3b2f8751e569c9 in term 1.
I20260923 15:15:50.222568 12360 catalog_manager.cc:1347] Generated new cluster ID: cc7d43256d93437290e98d9577f52670
I20260923 15:15:50.222613 12360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 15:15:50.223099 12358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:50.227029 12382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2f1b3ed6ec14a39bd3b2f8751e569c9. Latest consensus state: current_term: 1 leader_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.227115 12382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:50.227175 12385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.227223 12385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:50.220093 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.234018 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:50.235337 12381 catalog_manager.cc:1259] Loaded cluster ID: cc7d43256d93437290e98d9577f52670
I20260923 15:15:50.235387 12381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: loading cluster ID for follower catalog manager: success
W20260923 15:15:50.235530 12381 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260923 15:15:50.235571 12381 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:50.258015 12358 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f1b3ed6ec14a39bd3b2f8751e569c9" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 33087 } } peers { permanent_uuid: "859680366f7c4305b6993ae93a9485b6" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 33263 } } peers { permanent_uuid: "29a7f757c977405db7245bf123716320" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 45463 } } }
I20260923 15:15:50.258122 12358 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9 [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:50.259649 12360 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:50.260414 12360 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:50.270293 12360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2f1b3ed6ec14a39bd3b2f8751e569c9: Generated new TSK 0
I20260923 15:15:50.270498 12360 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 15:15:50.303001 12325 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.301331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.21.204.124:45463)
0923 15:15:50.301379 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.301381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.302296 (+ 915us) server_negotiation.cc:407] Connection header received
0923 15:15:50.302409 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.302410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.302422 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.302454 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.302916 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53584: BlockingRecv error: recv got EOF from 127.0.0.1:53584 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 15:15:50.303876 12174 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.301772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.21.204.126:33087)
0923 15:15:50.301814 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.301816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.303316 (+ 1500us) server_negotiation.cc:407] Connection header received
0923 15:15:50.303417 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.303418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.303429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.303460 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.303815 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39976: BlockingRecv error: recv got EOF from 127.0.0.1:39976 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 15:15:50.304744 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.302149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.21.204.125:33263)
0923 15:15:50.302191 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.302193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.304143 (+ 1950us) server_negotiation.cc:407] Connection header received
0923 15:15:50.304310 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.304311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.304322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.304351 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.304687 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43008: BlockingRecv error: recv got EOF from 127.0.0.1:43008 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 15:15:50.306844 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.306064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.21.204.125:33263)
0923 15:15:50.306109 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.306110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.306322 (+ 212us) server_negotiation.cc:407] Connection header received
0923 15:15:50.306427 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.306428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.306446 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.306475 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.306797 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43024: BlockingRecv error: recv got EOF from 127.0.0.1:43024 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 15:15:50.309839 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.309093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.21.204.125:33263)
0923 15:15:50.309126 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.309128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.309366 (+ 238us) server_negotiation.cc:407] Connection header received
0923 15:15:50.309461 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.309461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.309471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.309501 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.309796 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43034: BlockingRecv error: recv got EOF from 127.0.0.1:43034 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 15:15:50.313795 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.312956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.21.204.125:33263)
0923 15:15:50.313007 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.313010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.313294 (+ 284us) server_negotiation.cc:407] Connection header received
0923 15:15:50.313403 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.313404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.313415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.313450 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.313751 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43042: BlockingRecv error: recv got EOF from 127.0.0.1:43042 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260923 15:15:50.319790 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.318959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.21.204.125:33263)
0923 15:15:50.319023 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.319025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.319309 (+ 284us) server_negotiation.cc:407] Connection header received
0923 15:15:50.319407 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.319407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.319418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.319446 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.319748 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43048: BlockingRecv error: recv got EOF from 127.0.0.1:43048 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260923 15:15:50.329734 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.328919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.21.204.125:33263)
0923 15:15:50.328955 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.328957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.329217 (+ 260us) server_negotiation.cc:407] Connection header received
0923 15:15:50.329315 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.329316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.329326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.329355 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.329686 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43062: BlockingRecv error: recv got EOF from 127.0.0.1:43062 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 15:15:50.347892 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.347077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.21.204.125:33263)
0923 15:15:50.347118 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.347120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.347365 (+ 245us) server_negotiation.cc:407] Connection header received
0923 15:15:50.347460 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.347461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.347471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.347501 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.347834 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43066: BlockingRecv error: recv got EOF from 127.0.0.1:43066 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 15:15:50.382251 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.381546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.21.204.125:33263)
0923 15:15:50.381589 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.381591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.381622 (+ 31us) server_negotiation.cc:407] Connection header received
0923 15:15:50.381696 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.381697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.381709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.381786 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.382179 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43078: BlockingRecv error: recv got EOF from 127.0.0.1:43078 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 15:15:50.448359 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.447508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.21.204.125:33263)
0923 15:15:50.447613 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.447615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.447730 (+ 115us) server_negotiation.cc:407] Connection header received
0923 15:15:50.447837 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.447838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.447851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.447931 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.448286 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43080: BlockingRecv error: recv got EOF from 127.0.0.1:43080 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 15:15:50.578418 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.577571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.21.204.125:33263)
0923 15:15:50.577647 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.577650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.577767 (+ 117us) server_negotiation.cc:407] Connection header received
0923 15:15:50.577819 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.577820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.577833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.577913 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.578328 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43094: BlockingRecv error: recv got EOF from 127.0.0.1:43094 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260923 15:15:50.836220 12250 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:50.835564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.21.204.125:33263)
0923 15:15:50.835635 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:50.835637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:50.835659 (+ 22us) server_negotiation.cc:407] Connection header received
0923 15:15:50.835737 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:50.835738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:50.835748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:50.835800 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:50.836141 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35072: BlockingRecv error: recv got EOF from 127.0.0.1:35072 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 15:15:51.172662 12338 catalog_manager.cc:1259] Loaded cluster ID: cc7d43256d93437290e98d9577f52670
I20260923 15:15:51.172775 12338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: loading cluster ID for follower catalog manager: success
I20260923 15:15:51.173619 12338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: acquiring CA information for follower catalog manager: success
I20260923 15:15:51.173993 12338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29a7f757c977405db7245bf123716320: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 15:15:51.236785 12381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: acquiring CA information for follower catalog manager: success
I20260923 15:15:51.237182 12381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 859680366f7c4305b6993ae93a9485b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 15:15:51.352283 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:51.349350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.21.204.125:33263)
0923 15:15:51.349488 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:51.349490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:51.349538 (+ 48us) server_negotiation.cc:407] Connection header received
0923 15:15:51.349603 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:51.349603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:51.349654 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:51.349721 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:51.349906 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.350532 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.351310 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.351645 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.351717 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:51.351917 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:51.352041 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:51.352130 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:51.352167 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 15:15:51.352189 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260923 15:15:51.356791 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:51.353932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.21.204.124:45463)
0923 15:15:51.354053 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:51.354054 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:51.354062 (+ 8us) server_negotiation.cc:407] Connection header received
0923 15:15:51.354129 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:51.354130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:51.354164 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:51.354230 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:51.354314 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.354764 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.355913 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.356283 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.356368 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:51.356527 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:51.356620 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:51.356673 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:51.356715 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 15:15:51.356732 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260923 15:15:51.361292 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:51.358574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.21.204.125:33263)
0923 15:15:51.358617 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:51.358619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:51.358805 (+ 186us) server_negotiation.cc:407] Connection header received
0923 15:15:51.358912 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:51.358914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:51.358925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:51.358952 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:51.359124 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.359818 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.360549 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.360840 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.360905 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:51.361050 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:51.361080 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:51.361132 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:51.361194 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 15:15:51.361216 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:51.361438 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:51.358517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.21.204.124:45463)
0923 15:15:51.358559 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:51.358561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:51.358692 (+ 131us) server_negotiation.cc:407] Connection header received
0923 15:15:51.358756 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:51.358757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:51.358769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:51.358796 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:51.358979 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.359722 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.360704 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.360951 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.361013 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:51.361181 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:51.361213 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:51.361241 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:51.361329 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 15:15:51.361350 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:51.361747 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:51.358402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.21.204.126:33087)
0923 15:15:51.358523 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:51.358526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:51.358663 (+ 137us) server_negotiation.cc:407] Connection header received
0923 15:15:51.358751 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:51.358752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:51.358810 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:51.358893 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:51.359041 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.359635 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.360677 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:51.360990 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:51.361064 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:51.361308 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:51.361415 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:51.361460 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:51.361631 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 15:15:51.361656 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260923 15:15:51.363560 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 12103
I20260923 15:15:51.369282 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 12178
I20260923 15:15:51.374220 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 12253
2026-09-23T15:15:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2803 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 15:15:51 dist-test-slave-t5lt krb5kdc[12470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 15:15:51 dist-test-slave-t5lt krb5kdc[12470](info): set up 2 sockets
Sep 23 15:15:51 dist-test-slave-t5lt krb5kdc[12470](info): commencing operation
krb5kdc: starting...
W20260923 15:15:52.411507 22321 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/test-user.keytab.
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T15:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T15:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.204.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
Entry for principal kudu/127.21.204.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab.
I20260923 15:15:52.526449 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:38113
--webserver_interface=127.21.204.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:38113,127.21.204.125:34863,127.21.204.124:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 15:15:52.604756 12483 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 15:15:52.604998 12483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:52.605034 12483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:52.605077 12483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:52.606215 12483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:52.606278 12483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:52.606303 12483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:52.606329 12483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:52.606356 12483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:52.607802 12483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:38113,127.21.204.125:34863,127.21.204.124:33303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.126.keytab
--principal=kudu/127.21.204.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.126:38113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.204.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:52.608080 12483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:52.608295 12483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:52.613124 12488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:52.613082 12489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:52.613260 12483 server_base.cc:1034] running on GCE node
W20260923 15:15:52.613097 12491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:52.613525 12483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:52.613806 12483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:52.614970 12483 hybrid_clock.cc:648] HybridClock initialized: now 1790176552614958 us; error 61 us; skew 500 ppm
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:52.623490 12483 init.cc:377] Logged in from keytab as kudu/127.21.204.126@KRBTEST.COM (short username kudu)
I20260923 15:15:52.623960 12483 webserver.cc:460] Webserver started at http://127.21.204.126:33565/ using document root <none> and password file <none>
I20260923 15:15:52.624192 12483 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:52.624259 12483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:52.624411 12483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:52.625427 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data/instance:
uuid: "b55f55ace7404cea83443fd077c1263f"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "d24e2ffaeacfd37105af1ef96fb21b04"
server_key_iv: "f939c35c69825edb2f1fd26ee83c614a"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.625783 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/wal/instance:
uuid: "b55f55ace7404cea83443fd077c1263f"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "d24e2ffaeacfd37105af1ef96fb21b04"
server_key_iv: "f939c35c69825edb2f1fd26ee83c614a"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.627099 12483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:52.627847 12498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:52.628043 12483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:52.628144 12483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/wal
uuid: "b55f55ace7404cea83443fd077c1263f"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "d24e2ffaeacfd37105af1ef96fb21b04"
server_key_iv: "f939c35c69825edb2f1fd26ee83c614a"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.628224 12483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:52.655750 12483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:52.657636 12483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:52.657800 12483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:52.662083 12483 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.126:38113
I20260923 15:15:52.662130 12550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.126:38113 every 8 connection(s)
I20260923 15:15:52.662461 12483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/data/info.pb
I20260923 15:15:52.663281 12551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:52.664798 12551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:52.671131 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 12483
I20260923 15:15:52.671231 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-0/wal/instance
I20260923 15:15:52.671452 22321 external_mini_cluster.cc:1428] Setting key f86405d0c0e5f95b2f8534d34598312e
I20260923 15:15:52.674539 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.665535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.21.204.126:38113)
0923 15:15:52.665699 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.665708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.665824 (+ 116us) server_negotiation.cc:407] Connection header received
0923 15:15:52.666405 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.666407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.666423 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.666504 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.667020 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.667816 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.668711 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.669035 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.669121 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.671013 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.671022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.671023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.671029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.672880 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.673371 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.673373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.673374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.673459 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.673719 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.673720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.673720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.673996 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.674062 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.674281 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.674377 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260923 15:15:52.674671 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.665519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:38113 (local address 127.0.0.1:49382)
0923 15:15:52.665696 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.665734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:52.666257 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:52.666531 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:52.666533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:52.666763 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:52.666927 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.666931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.667908 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.667909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.668619 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.668621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.669114 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.669115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.669172 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.669578 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:52.669586 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:52.670899 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:52.672972 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.672980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.672983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.673269 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.673541 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.673542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.673542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.673653 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.674065 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:52.674072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:52.674144 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 15:15:52.674244 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 15:15:52.674485 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":59,"thread_start_us":94,"threads_started":1}
I20260923 15:15:52.675371 12551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:15:52.675716 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.675594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60240)
0923 15:15:52.675631 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.675683 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260923 15:15:52.675856 12500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
I20260923 15:15:52.676322 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.676209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60244)
0923 15:15:52.676262 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.676293 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 15:15:52.676458 12551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:34863: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.204.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab.
Entry for principal kudu/127.21.204.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab.
I20260923 15:15:52.696163 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:34863
--webserver_interface=127.21.204.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab
--principal=kudu/127.21.204.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:38113,127.21.204.125:34863,127.21.204.124:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:15:52.725700 12551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } attempt: 1
I20260923 15:15:52.726240 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.726086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60256)
0923 15:15:52.726155 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.726208 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 15:15:52.726612 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.726509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60270)
0923 15:15:52.726559 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.726592 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 15:15:52.726711 12551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:34863: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
W20260923 15:15:52.776430 12558 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 15:15:52.776669 12558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:52.776710 12558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:52.776746 12558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:52.777861 12558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:52.777971 12558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:52.778012 12558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:52.778041 12558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:52.778071 12558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:52.779505 12558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:38113,127.21.204.125:34863,127.21.204.124:33303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.125.keytab
--principal=kudu/127.21.204.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.125:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.204.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:52.779780 12558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:52.780004 12558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:52.784804 12563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:52.784865 12566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:52.784816 12564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:52.784893 12558 server_base.cc:1034] running on GCE node
I20260923 15:15:52.785255 12558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:52.785545 12558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:52.786685 12558 hybrid_clock.cc:648] HybridClock initialized: now 1790176552786662 us; error 38 us; skew 500 ppm
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:52.789165 12558 init.cc:377] Logged in from keytab as kudu/127.21.204.125@KRBTEST.COM (short username kudu)
I20260923 15:15:52.789603 12558 webserver.cc:460] Webserver started at http://127.21.204.125:45555/ using document root <none> and password file <none>
I20260923 15:15:52.789824 12558 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:52.789913 12558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:52.790086 12558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:52.791109 12558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data/instance:
uuid: "c2c4c4d4cc8846e29b0363278cd095b0"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "40939a24693ef39d6c46b872c3e60dec"
server_key_iv: "862e7af815128fcd45a12fe329e6b5be"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.791469 12558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/wal/instance:
uuid: "c2c4c4d4cc8846e29b0363278cd095b0"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "40939a24693ef39d6c46b872c3e60dec"
server_key_iv: "862e7af815128fcd45a12fe329e6b5be"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.792786 12558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 15:15:52.793594 12573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:52.793793 12558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 15:15:52.793910 12558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/wal
uuid: "c2c4c4d4cc8846e29b0363278cd095b0"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "40939a24693ef39d6c46b872c3e60dec"
server_key_iv: "862e7af815128fcd45a12fe329e6b5be"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.793987 12558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:52.794854 12551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } attempt: 2
I20260923 15:15:52.795241 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.795102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60276)
0923 15:15:52.795165 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.795205 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 15:15:52.795591 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.795493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60284)
0923 15:15:52.795539 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.795567 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 15:15:52.795698 12551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.125:34863: Network error: Client connection negotiation failed: client connection to 127.21.204.125:34863: connect: Connection refused (error 111)
I20260923 15:15:52.805275 12558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:52.807109 12558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:52.807260 12558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:52.811187 12558 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.125:34863
I20260923 15:15:52.811257 12625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.125:34863 every 8 connection(s)
I20260923 15:15:52.811573 12558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/data/info.pb
I20260923 15:15:52.812417 12626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:52.813930 12626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:52.820457 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 12558
I20260923 15:15:52.820591 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-1/wal/instance
I20260923 15:15:52.820773 22321 external_mini_cluster.cc:1428] Setting key 6ab9b00e4314d9b7466c9258e9cc27c6
I20260923 15:15:52.822600 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.814624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.21.204.126:38113)
0923 15:15:52.814682 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.814684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.814899 (+ 215us) server_negotiation.cc:407] Connection header received
0923 15:15:52.815349 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.815351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.815364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.815427 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.815970 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.816466 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.817351 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.817558 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.817607 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.819487 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.819489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.819489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.819495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.821001 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.821466 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.821467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.821467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.821516 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.821820 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.821820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.821821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.821985 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.822038 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.822375 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.822470 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 15:15:52.822691 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.814609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:38113 (local address 127.0.0.1:49388)
0923 15:15:52.814778 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.814813 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:52.815264 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:52.815496 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:52.815499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:52.815717 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:52.815894 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.815897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.816548 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.816548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.817278 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.817280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.817626 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.817626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.817721 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.818163 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:52.818171 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:52.819405 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:52.821076 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.821083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.821086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.821391 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.821581 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.821582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.821582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.821748 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.822047 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:52.822054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:52.822224 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 15:15:52.822340 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 15:15:52.822514 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260923 15:15:52.823220 12626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.204.124@KRBTEST.COM; defaulting to no policy
I20260923 15:15:52.830495 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.823445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60292)
0923 15:15:52.823492 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.823501 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:52.823540 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:52.823789 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:52.823790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:52.823917 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:52.823955 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.823957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.824853 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.824854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.825497 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.825500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.826014 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.826015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.826105 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.826427 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:52.826432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:52.827713 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:52.829459 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.829460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.829460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.829620 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.829837 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.829839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.829839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.829913 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.830189 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:52.830190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:52.830225 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 15:15:52.830296 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 15:15:52.830388 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 15:15:52.830579 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.823532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.21.204.125:34863)
0923 15:15:52.823647 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.823650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.823664 (+ 14us) server_negotiation.cc:407] Connection header received
0923 15:15:52.823678 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.823679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.823688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.823776 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.824042 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.824756 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.825587 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.825913 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.825992 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.827812 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.827820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.827821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.827827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.829356 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.829713 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.829716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.829716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.829772 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.829966 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.829967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.829967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.830116 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.830211 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.830326 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.830446 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":94,"thread_start_us":75,"threads_started":1}
I20260923 15:15:52.831223 12626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.204.124@KRBTEST.COM" created.
I20260923 15:15:52.831547 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.831422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45682)
0923 15:15:52.831472 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.831518 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 15:15:52.831686 12575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
I20260923 15:15:52.832068 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.831984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45688)
0923 15:15:52.832025 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.832050 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 15:15:52.832195 12626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:33303: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.204.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab.
Entry for principal kudu/127.21.204.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab.
I20260923 15:15:52.845696 22321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOKY8g3/build/release/bin/kudu
/tmp/dist-test-taskOKY8g3/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:33303
--webserver_interface=127.21.204.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab
--principal=kudu/127.21.204.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.204.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.204.126:38113,127.21.204.125:34863,127.21.204.124:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 15:15:52.881436 12626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } attempt: 1
I20260923 15:15:52.881958 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.881771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45696)
0923 15:15:52.881835 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.881916 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 15:15:52.882301 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.882202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45706)
0923 15:15:52.882245 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.882271 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 15:15:52.882406 12626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:33303: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
I20260923 15:15:52.886850 12551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } attempt: 3
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
I20260923 15:15:52.894443 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.887173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60294)
0923 15:15:52.887230 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.887242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:52.887307 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:52.887435 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:52.887435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:52.887601 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:52.887668 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.887670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.888357 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.888359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.888948 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.888951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.889668 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.889670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.889755 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.890275 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:52.890280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:52.891634 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:52.893334 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.893335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.893336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.893553 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.893748 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.893749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.893750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.893819 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.894104 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:52.894105 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:52.894157 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 15:15:52.894232 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 15:15:52.894346 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 15:15:52.894429 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.887244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.21.204.125:34863)
0923 15:15:52.887307 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.887309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.887320 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:15:52.887370 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.887371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.887382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.887432 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.887757 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.888272 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.889036 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.889543 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.889665 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.891722 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.891723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.891724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.891732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.893201 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.893626 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.893627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.893628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.893681 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.893927 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.893928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.893928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.894036 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.894091 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.894257 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.894349 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 15:15:52.894876 12551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:15:52.895239 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.895129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45718)
0923 15:15:52.895177 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.895212 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 15:15:52.895555 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.895466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45730)
0923 15:15:52.895511 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.895537 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 15:15:52.895648 12551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:33303: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
I20260923 15:15:52.926793 12551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } attempt: 1
I20260923 15:15:52.927290 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.927143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45736)
0923 15:15:52.927210 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.927253 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 15:15:52.927700 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.927589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45746)
0923 15:15:52.927641 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.927674 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 15:15:52.927806 12551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:33303: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
W20260923 15:15:52.928575 12633 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 15:15:52.928754 12633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 15:15:52.928771 12633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 15:15:52.928834 12633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 15:15:52.930126 12633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 15:15:52.930166 12633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 15:15:52.930179 12633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 15:15:52.930192 12633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 15:15:52.930236 12633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 15:15:52.931758 12633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.204.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.204.126:38113,127.21.204.125:34863,127.21.204.124:33303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/krb5kdc/kudu_127.21.204.124.keytab
--principal=kudu/127.21.204.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.204.124:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.204.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 14:06:54 UTC on ccycloud.quasar-athtfy.root.comops.site
build id 4667496
I20260923 15:15:52.931995 12633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 15:15:52.932224 12633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 15:15:52.936924 12641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 15:15:52.936903 12639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:52.937021 12633 server_base.cc:1034] running on GCE node
W20260923 15:15:52.936902 12638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 15:15:52.937355 12633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 15:15:52.937702 12633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 15:15:52.938875 12633 hybrid_clock.cc:648] HybridClock initialized: now 1790176552938858 us; error 29 us; skew 500 ppm
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 15:15:52.941262 12633 init.cc:377] Logged in from keytab as kudu/127.21.204.124@KRBTEST.COM (short username kudu)
I20260923 15:15:52.941674 12633 webserver.cc:460] Webserver started at http://127.21.204.124:37895/ using document root <none> and password file <none>
I20260923 15:15:52.941936 12633 fs_manager.cc:359] Metadata directory not provided
I20260923 15:15:52.941984 12633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 15:15:52.942121 12633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 15:15:52.943138 12633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data/instance:
uuid: "97acc82a72714e6689983f6b645172d7"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "6043f9cf58c7be94766c725a63202cbb"
server_key_iv: "edbc03de0a45f7784c7315cd94028685"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.943485 12633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/wal/instance:
uuid: "97acc82a72714e6689983f6b645172d7"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "6043f9cf58c7be94766c725a63202cbb"
server_key_iv: "edbc03de0a45f7784c7315cd94028685"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.944813 12633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 15:15:52.945622 12648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 15:15:52.945811 12633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 15:15:52.945928 12633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data,/tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/wal
uuid: "97acc82a72714e6689983f6b645172d7"
format_stamp: "Formatted at 2026-09-23 15:15:52 on dist-test-slave-t5lt"
server_key: "6043f9cf58c7be94766c725a63202cbb"
server_key_iv: "edbc03de0a45f7784c7315cd94028685"
server_key_version: "encryptionkey@0"
I20260923 15:15:52.946012 12633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 15:15:52.950541 12626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } attempt: 2
I20260923 15:15:52.951007 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.950882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45748)
0923 15:15:52.950939 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.950976 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 15:15:52.951331 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.951254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45764)
0923 15:15:52.951290 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.951314 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260923 15:15:52.951464 12626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.204.124:33303: Network error: Client connection negotiation failed: client connection to 127.21.204.124:33303: connect: Connection refused (error 111)
I20260923 15:15:52.953572 12633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 15:15:52.955423 12633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 15:15:52.955576 12633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 15:15:52.959483 12633 rpc_server.cc:307] RPC server started. Bound to: 127.21.204.124:33303
I20260923 15:15:52.959539 12700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.204.124:33303 every 8 connection(s)
I20260923 15:15:52.959861 12633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/data/info.pb
I20260923 15:15:52.960784 12701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 15:15:52.962307 12701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:52.969847 22321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOKY8g3/build/release/bin/kudu as pid 12633
I20260923 15:15:52.969998 22321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOKY8g3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790176278407460-22321-0/minicluster-data/master-2/wal/instance
I20260923 15:15:52.970227 22321 external_mini_cluster.cc:1428] Setting key 4a69d3e572ed94be5c465870490a0691
I20260923 15:15:52.971197 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.962984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.21.204.126:38113)
0923 15:15:52.963043 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.963045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.963174 (+ 129us) server_negotiation.cc:407] Connection header received
0923 15:15:52.963635 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.963636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.963647 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.963704 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.964265 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.964733 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.965661 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.965870 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.965993 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.968080 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.968081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.968081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.968088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.969585 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.970141 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.970142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.970143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.970208 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.970491 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.970492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.970492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.970647 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.970718 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.971026 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.971117 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 15:15:52.971369 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.962905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.126:38113 (local address 127.0.0.1:49398)
0923 15:15:52.963058 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.963097 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:52.963538 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:52.963778 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:52.963781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:52.964009 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:52.964188 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.964191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.964816 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.964816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.965572 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.965574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.966015 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.966016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.966132 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.966633 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:52.966642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:52.967980 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:52.969696 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.969707 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.969709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.970025 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.970278 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.970280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.970280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.970397 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.970748 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:52.970753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:52.970878 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 15:15:52.970972 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 15:15:52.971134 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260923 15:15:52.971964 12701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:15:52.979077 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.972226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.125:34863 (local address 127.0.0.1:60302)
0923 15:15:52.972347 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.972357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:52.972438 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:52.972553 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:52.972554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:52.972695 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:52.972746 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.972747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.973330 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.973331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.973824 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.973826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.974237 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.974239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.974309 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.974807 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:52.974812 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:52.976135 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:52.977915 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.977917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.977917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.978195 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.978452 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.978452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.978453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.978519 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.978808 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:52.978808 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:52.978859 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 15:15:52.978932 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 15:15:52.979000 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 15:15:52.979197 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.972298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.21.204.125:34863)
0923 15:15:52.972341 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.972343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.972424 (+ 81us) server_negotiation.cc:407] Connection header received
0923 15:15:52.972496 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.972497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.972506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.972549 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.972815 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.973248 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.973951 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.974155 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.974237 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.976244 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.976246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.976246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.976254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.977814 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.978290 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.978291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.978291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.978372 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.978591 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.978592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.978592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.978704 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.978816 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.978967 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.979051 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:52.979476 12701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
I20260923 15:15:52.980098 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.970629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.21.204.124:33303)
0923 15:15:52.970918 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.970921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.970937 (+ 16us) server_negotiation.cc:407] Connection header received
0923 15:15:52.970975 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.970977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.970992 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.971065 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.971930 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.972963 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.973710 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.974118 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.974201 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.976734 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.976743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.976743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.976751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.978707 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.979172 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.979176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.979176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.979235 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.979490 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.979491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.979491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.979673 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.979843 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.979968 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.980034 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":75,"threads_started":1}
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.124@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.126@KRBTEST.COM
I20260923 15:15:52.986727 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.979816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.21.204.124:33303)
0923 15:15:52.980196 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.980198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.980205 (+ 7us) server_negotiation.cc:407] Connection header received
0923 15:15:52.980223 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.980224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.980234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.980281 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.980532 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.981142 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.981804 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.982098 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.982195 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.983902 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.983904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.983904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.983912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.985282 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.985645 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.985646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.985646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.985709 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.985955 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.985956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.985956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.986068 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.986127 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.986583 (+ 456us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.986674 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":366,"thread_start_us":114,"threads_started":1}
I20260923 15:15:52.986972 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.979714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45792)
0923 15:15:52.979786 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:52.979795 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:52.979828 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:52.980283 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:52.980284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:52.980420 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:52.980452 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.980454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.981240 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.981241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.981705 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:52.981706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.982196 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.982197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:52.982247 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.982631 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:52.982635 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:52.983810 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:52.985364 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.985366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.985366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.985568 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.985791 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:52.985791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:52.985792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:52.985869 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:52.986398 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:52.986399 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:52.986447 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 15:15:52.986532 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 15:15:52.986627 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 15:15:52.988653 12701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: Bootstrap starting.
I20260923 15:15:52.988852 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.981012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.21.204.126:38113)
0923 15:15:52.981060 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.981062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.981086 (+ 24us) server_negotiation.cc:407] Connection header received
0923 15:15:52.981108 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.981109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.981117 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.981172 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.981706 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.982217 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.983061 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.983261 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.983344 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.985913 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.985914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.985915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.985922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.987687 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.987998 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.987999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.987999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.988059 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.988241 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.988243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.988243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.988380 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.988567 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.988606 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.988771 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:15:52.989775 12701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:52.990159 12701 log.cc:824] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:52.990882 12701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: No bootstrap required, opened a new log
I20260923 15:15:52.992439 12701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:52.992601 12701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97acc82a72714e6689983f6b645172d7, State: Initialized, Role: FOLLOWER
I20260923 15:15:52.992697 12701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:52.992982 12708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:52.993034 12701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:52.993086 12708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [sys.catalog]: This master's current role is: FOLLOWER
Sep 23 15:15:52 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.204.125@KRBTEST.COM
W20260923 15:15:52.995007 12719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:15:52.995105 12719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:52.995472 12633 master_runner.cc:186] Error getting master registration for 127.21.204.126:38113: OK, instance_id { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" instance_seqno: 1790176552660785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 15:15:52.997941 12633 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 15:15:52.998050 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:52.989681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.21.204.125:34863)
0923 15:15:52.989731 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:52.989732 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:52.989767 (+ 35us) server_negotiation.cc:407] Connection header received
0923 15:15:52.989973 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:52.989974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:52.989984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:52.990038 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:52.990572 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.991031 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.991674 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:52.991892 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:52.991969 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:52.994550 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:52.994552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:52.994552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:52.994560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:52.996240 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.996861 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.996862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.996862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.996938 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:52.997143 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:52.997144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:52.997144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:52.997275 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:52.997433 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:52.997859 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 15:15:52.997966 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 15:15:52.999719 22321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 15:15:53.000671 12525 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b55f55ace7404cea83443fd077c1263f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49406
I20260923 15:15:53.002914 12551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } attempt: 2
Sep 23 15:15:53 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.126@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:15:53.011003 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:53.003136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45804)
0923 15:15:53.003175 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:53.003225 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:53.003273 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:53.003436 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:53.003437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:53.003611 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:53.003652 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:53.003653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.004594 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.004595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:53.005372 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:53.005374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.005836 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.005836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:53.005909 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:53.006321 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:53.006326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:53.007850 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:53.009797 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:53.009798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:53.009798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:53.010039 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:53.010276 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:53.010277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:53.010277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:53.010344 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:53.010683 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:53.010684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:53.010728 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 15:15:53.010796 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 15:15:53.010893 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 15:15:53.011019 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:53.003202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.21.204.124:33303)
0923 15:15:53.003243 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.003245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.003256 (+ 11us) server_negotiation.cc:407] Connection header received
0923 15:15:53.003351 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.003352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.003362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.003432 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.003726 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.004512 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.005458 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.005757 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.005828 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:53.007953 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:53.007954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:53.007955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:53.007962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:53.009712 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.010121 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.010122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.010122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.010183 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.010443 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.010443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.010444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.010611 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:53.010683 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:53.010840 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 15:15:53.010931 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:53.012564 12551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f: Bootstrap starting.
I20260923 15:15:53.013298 12551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:53.013592 12551 log.cc:824] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:53.014477 12551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f: No bootstrap required, opened a new log
I20260923 15:15:53.015815 12551 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:53.015975 12551 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55f55ace7404cea83443fd077c1263f, State: Initialized, Role: FOLLOWER
I20260923 15:15:53.016059 12551 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:53.016358 12551 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:53.016397 12525 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 15:15:53.016326 12722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.016466 12525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:53.016551 12722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:53.017320 12525 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:53.017947 12675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b55f55ace7404cea83443fd077c1263f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "97acc82a72714e6689983f6b645172d7"
I20260923 15:15:53.017710 12525 leader_election.cc:288] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [CANDIDATE]: Term 1 election: Requested vote from peers c2c4c4d4cc8846e29b0363278cd095b0 (127.21.204.125:34863), 97acc82a72714e6689983f6b645172d7 (127.21.204.124:33303)
I20260923 15:15:53.018070 12675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [term 0 FOLLOWER]: Advancing to term 1
W20260923 15:15:53.018599 12733 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:15:53.018702 12733 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:53.018975 12675 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b55f55ace7404cea83443fd077c1263f in term 1.
I20260923 15:15:53.020002 12500 leader_election.cc:302] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97acc82a72714e6689983f6b645172d7, b55f55ace7404cea83443fd077c1263f; no voters:
I20260923 15:15:53.020126 12722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 15:15:53.020248 12722 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [term 1 LEADER]: Becoming Leader. State: Replica: b55f55ace7404cea83443fd077c1263f, State: Running, Role: LEADER
I20260923 15:15:53.020332 12722 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:53.020723 12722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55f55ace7404cea83443fd077c1263f. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.020821 12722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:53.019050 12483 master_runner.cc:186] Error getting master registration for 127.21.204.125:34863: OK, instance_id { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" instance_seqno: 1790176552810085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 15:15:53.021795 12483 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 15:15:53.026489 12741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 15:15:53.026700 12741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 15:15:53.029152 12675 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [term 1 FOLLOWER]: Refusing update from remote peer b55f55ace7404cea83443fd077c1263f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:53.029500 12739 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:53.030840 12708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55f55ace7404cea83443fd077c1263f. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.030943 12708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:53.039212 12722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.039330 12722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:53.039402 12708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.039482 12708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:53.039711 12745 mvcc.cc:205] Tried to move back new op lower bound from 7332563161203855360 to 7332563161172008960. Current Snapshot: MvccSnapshot[applied={T|T < 7332563161203855360}]
I20260923 15:15:53.042589 12626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } attempt: 3
I20260923 15:15:53.043781 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:53.021712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.21.204.124:33303)
0923 15:15:53.021749 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.021752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.029576 (+ 7824us) server_negotiation.cc:407] Connection header received
0923 15:15:53.030393 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.030395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.030405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.030435 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.034248 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.035563 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.036940 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.037249 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.037287 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:53.039841 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:53.039842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:53.039843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:53.039850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:53.041637 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.042479 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.042480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.042480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.042540 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.043009 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.043010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.043011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.043154 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:53.043244 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:53.043617 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 15:15:53.043691 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:15:53.045394 12741 catalog_manager.cc:1347] Generated new cluster ID: 335e4deec12c49c7a22ce90e5cc636c6
I20260923 15:15:53.045405 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:53.021421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.21.204.125:34863)
0923 15:15:53.021463 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.021465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.028925 (+ 7460us) server_negotiation.cc:407] Connection header received
0923 15:15:53.029404 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.029405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.029419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.029449 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.031349 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.032095 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.034508 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.034814 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.034856 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:53.040137 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:53.040138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:53.040138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:53.040146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:53.042385 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.043957 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.043958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.043958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.044018 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.044683 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.044684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.044684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.044813 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:53.044902 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:53.045236 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 15:15:53.045315 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:53.045442 12741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 15:15:53 dist-test-slave-t5lt krb5kdc[12470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790176552, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.204.125@KRBTEST.COM for kudu/127.21.204.124@KRBTEST.COM
I20260923 15:15:53.049851 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:53.021215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.21.204.126:38113)
0923 15:15:53.021258 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.021260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.027999 (+ 6739us) server_negotiation.cc:407] Connection header received
0923 15:15:53.028099 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.028100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.028111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.028147 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.031242 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.031987 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.033249 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.033561 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.033627 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:53.036076 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:53.036077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:53.036078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:53.036085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:53.038261 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.044239 (+ 5978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.044240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.044240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.044292 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.049250 (+ 4958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.049251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.049252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.049388 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:53.049479 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:53.049674 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 15:15:53.049764 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 15:15:53.052250 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:53.042831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.204.124:33303 (local address 127.0.0.1:45820)
0923 15:15:53.042876 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 15:15:53.042887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 15:15:53.042919 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 15:15:53.043098 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 15:15:53.043099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 15:15:53.043261 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 15:15:53.043306 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:53.043308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.044268 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.044269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:53.045058 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 15:15:53.045061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.045557 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.045558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 15:15:53.045612 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:53.046051 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 15:15:53.046055 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 15:15:53.047486 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 15:15:53.050877 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:53.050878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:53.050879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:53.051105 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:53.051439 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 15:15:53.051440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 15:15:53.051441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 15:15:53.051499 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 15:15:53.051957 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 15:15:53.051971 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 15:15:53.052015 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 15:15:53.052060 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 15:15:53.052153 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 15:15:53.053067 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:53.042871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.21.204.124:33303)
0923 15:15:53.042928 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.042929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.042938 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:53.042961 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.042962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.042973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.043039 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.043385 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.044168 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.045156 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:53.045457 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:53.045978 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:53.047642 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 15:15:53.047644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 15:15:53.047644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 15:15:53.047651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 15:15:53.050766 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.051238 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.051239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.051240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.051328 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 15:15:53.051599 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 15:15:53.051599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 15:15:53.051600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 15:15:53.051846 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 15:15:53.052598 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:53.052719 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 15:15:53.052860 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 15:15:53.054821 12626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: Bootstrap starting.
I20260923 15:15:53.055478 12626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: Neither blocks nor log segments found. Creating new log.
I20260923 15:15:53.055758 12626 log.cc:824] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: Log is configured to *not* fsync() on all Append() calls
I20260923 15:15:53.066584 12626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: No bootstrap required, opened a new log
I20260923 15:15:53.068615 12626 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:53.068747 12626 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c4c4d4cc8846e29b0363278cd095b0, State: Initialized, Role: FOLLOWER
I20260923 15:15:53.068818 12626 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } }
I20260923 15:15:53.069156 12626 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 15:15:53.070755 12600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b55f55ace7404cea83443fd077c1263f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2c4c4d4cc8846e29b0363278cd095b0"
I20260923 15:15:53.070667 12599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 15:15:53.071893 12599 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [term 1 FOLLOWER]: Refusing update from remote peer b55f55ace7404cea83443fd077c1263f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 15:15:53.072352 12722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 15:15:53.074101 12751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.074182 12751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:53.074221 12751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55f55ace7404cea83443fd077c1263f. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.074254 12751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:53.074278 12751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.074309 12751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 15:15:53.078384 12764 mvcc.cc:205] Tried to move back new op lower bound from 7332563161203855360 to 7332563161172008960. Current Snapshot: MvccSnapshot[applied={T|T < 7332563161203855360}]
W20260923 15:15:53.078481 12762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 15:15:53.078996 12762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 15:15:53.079108 12600 raft_consensus.cc:2396] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b55f55ace7404cea83443fd077c1263f for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260923 15:15:53.098856 12722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b55f55ace7404cea83443fd077c1263f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b55f55ace7404cea83443fd077c1263f" member_type: VOTER last_known_addr { host: "127.21.204.126" port: 38113 } } peers { permanent_uuid: "c2c4c4d4cc8846e29b0363278cd095b0" member_type: VOTER last_known_addr { host: "127.21.204.125" port: 34863 } } peers { permanent_uuid: "97acc82a72714e6689983f6b645172d7" member_type: VOTER last_known_addr { host: "127.21.204.124" port: 33303 } } }
I20260923 15:15:53.098986 12722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f [sys.catalog]: This master's current role is: LEADER
I20260923 15:15:53.100924 12741 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 15:15:53.101656 12741 catalog_manager.cc:1504] Loading token signing keys...
I20260923 15:15:53.109974 12741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b55f55ace7404cea83443fd077c1263f: Generated new TSK 0
I20260923 15:15:53.110177 12741 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 15:15:53.134091 12553 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.132440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.21.204.126:38113)
0923 15:15:53.132484 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.132487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.133361 (+ 874us) server_negotiation.cc:407] Connection header received
0923 15:15:53.133479 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.133481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.133494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.133526 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.134006 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49432: BlockingRecv error: recv got EOF from 127.0.0.1:49432 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 15:15:53.135887 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.132795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.21.204.125:34863)
0923 15:15:53.132837 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.132839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.134326 (+ 1487us) server_negotiation.cc:407] Connection header received
0923 15:15:53.135466 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.135468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.135480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.135510 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.135825 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60334: BlockingRecv error: recv got EOF from 127.0.0.1:60334 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 15:15:53.136606 12706 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.133097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.21.204.124:33303)
0923 15:15:53.133150 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.133152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.136122 (+ 2970us) server_negotiation.cc:407] Connection header received
0923 15:15:53.136218 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.136219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.136230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.136258 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.136547 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45826: BlockingRecv error: recv got EOF from 127.0.0.1:45826 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260923 15:15:53.138969 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.137817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.21.204.125:34863)
0923 15:15:53.137851 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.137852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.138467 (+ 615us) server_negotiation.cc:407] Connection header received
0923 15:15:53.138574 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.138575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.138594 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.138624 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.138922 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60336: BlockingRecv error: recv got EOF from 127.0.0.1:60336 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 15:15:53.141983 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.141213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.21.204.125:34863)
0923 15:15:53.141244 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.141245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.141479 (+ 234us) server_negotiation.cc:407] Connection header received
0923 15:15:53.141573 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.141573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.141583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.141612 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.141929 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60352: BlockingRecv error: recv got EOF from 127.0.0.1:60352 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 15:15:53.145848 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.145096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.21.204.125:34863)
0923 15:15:53.145130 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.145132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.145341 (+ 209us) server_negotiation.cc:407] Connection header received
0923 15:15:53.145442 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.145443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.145456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.145488 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.145811 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60364: BlockingRecv error: recv got EOF from 127.0.0.1:60364 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 15:15:53.151670 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.150914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.21.204.125:34863)
0923 15:15:53.150947 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.150949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.151188 (+ 239us) server_negotiation.cc:407] Connection header received
0923 15:15:53.151282 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.151283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.151292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.151321 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.151630 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60374: BlockingRecv error: recv got EOF from 127.0.0.1:60374 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 15:15:53.161644 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.160814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.21.204.125:34863)
0923 15:15:53.160849 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.160851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.161108 (+ 257us) server_negotiation.cc:407] Connection header received
0923 15:15:53.161208 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.161209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.161219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.161247 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.161594 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60388: BlockingRecv error: recv got EOF from 127.0.0.1:60388 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 15:15:53.179643 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.178893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.21.204.125:34863)
0923 15:15:53.178937 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.178939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.179083 (+ 144us) server_negotiation.cc:407] Connection header received
0923 15:15:53.179155 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.179157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.179168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.179243 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.179573 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60400: BlockingRecv error: recv got EOF from 127.0.0.1:60400 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 15:15:53.213547 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.212795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.21.204.125:34863)
0923 15:15:53.212849 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.212852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.212991 (+ 139us) server_negotiation.cc:407] Connection header received
0923 15:15:53.213066 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.213067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.213079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.213153 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.213477 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60404: BlockingRecv error: recv got EOF from 127.0.0.1:60404 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260923 15:15:53.279459 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.278767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.21.204.125:34863)
0923 15:15:53.278863 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.278872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.278938 (+ 66us) server_negotiation.cc:407] Connection header received
0923 15:15:53.278993 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.278994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.279005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.279057 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.279399 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60418: BlockingRecv error: recv got EOF from 127.0.0.1:60418 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 15:15:53.409415 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.408623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.21.204.125:34863)
0923 15:15:53.408710 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.408712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.408783 (+ 71us) server_negotiation.cc:407] Connection header received
0923 15:15:53.408857 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.408858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.408871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.408945 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.409360 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60420: BlockingRecv error: recv got EOF from 127.0.0.1:60420 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 15:15:53.667361 12630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 15:15:53.666725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.21.204.125:34863)
0923 15:15:53.666809 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:53.666811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:53.666857 (+ 46us) server_negotiation.cc:407] Connection header received
0923 15:15:53.666883 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:53.666884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:53.666894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:53.666939 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 15:15:53.667261 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60424: BlockingRecv error: recv got EOF from 127.0.0.1:60424 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 15:15:53.995597 12719 catalog_manager.cc:1259] Loaded cluster ID: 335e4deec12c49c7a22ce90e5cc636c6
I20260923 15:15:53.995671 12719 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: loading cluster ID for follower catalog manager: success
I20260923 15:15:53.996649 12719 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: acquiring CA information for follower catalog manager: success
I20260923 15:15:53.996979 12719 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97acc82a72714e6689983f6b645172d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 15:15:54.079509 12762 catalog_manager.cc:1259] Loaded cluster ID: 335e4deec12c49c7a22ce90e5cc636c6
I20260923 15:15:54.079591 12762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: loading cluster ID for follower catalog manager: success
I20260923 15:15:54.080421 12762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: acquiring CA information for follower catalog manager: success
I20260923 15:15:54.080778 12762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2c4c4d4cc8846e29b0363278cd095b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 15:15:54.183202 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:54.180543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.21.204.125:34863)
0923 15:15:54.180686 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:54.180687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:54.180700 (+ 13us) server_negotiation.cc:407] Connection header received
0923 15:15:54.180770 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:54.180770 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:54.180824 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:54.180914 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:54.181055 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.181578 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.182319 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.182540 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.182629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:54.182827 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:54.182939 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:54.183034 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:54.183073 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 15:15:54.183096 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260923 15:15:54.187419 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:54.184990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.21.204.124:33303)
0923 15:15:54.185094 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:54.185097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:54.185106 (+ 9us) server_negotiation.cc:407] Connection header received
0923 15:15:54.185154 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:54.185155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:54.185199 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:54.185283 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:54.185372 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.185854 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.186613 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.186819 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.186903 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:54.187080 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:54.187178 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:54.187260 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:54.187301 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 15:15:54.187323 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260923 15:15:54.192248 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:54.189056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.21.204.126:38113)
0923 15:15:54.189173 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:54.189175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:54.189277 (+ 102us) server_negotiation.cc:407] Connection header received
0923 15:15:54.189408 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:54.189410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:54.189465 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:54.189525 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:54.189754 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.190555 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.191336 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.191649 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.191716 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:54.191912 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:54.192020 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:54.192083 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:54.192129 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 15:15:54.192157 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260923 15:15:54.192333 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:54.189397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.21.204.124:33303)
0923 15:15:54.189433 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:54.189434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:54.189539 (+ 105us) server_negotiation.cc:407] Connection header received
0923 15:15:54.189635 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:54.189636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:54.189645 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:54.189695 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:54.189834 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.190348 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.191385 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.191686 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.191743 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:54.192036 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:54.192070 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:54.192119 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:54.192214 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 15:15:54.192239 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 15:15:54.192402 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 15:15:54.189203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.21.204.125:34863)
0923 15:15:54.189244 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 15:15:54.189246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 15:15:54.189316 (+ 70us) server_negotiation.cc:407] Connection header received
0923 15:15:54.189445 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 15:15:54.189446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 15:15:54.189458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 15:15:54.189508 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 15:15:54.189740 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.190511 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.191547 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 15:15:54.191816 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 15:15:54.191876 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 15:15:54.192050 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 15:15:54.192083 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 15:15:54.192130 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 15:15:54.192285 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 15:15:54.192313 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 15:15:54.194438 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 12483
I20260923 15:15:54.200255 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 12558
I20260923 15:15:54.205343 22321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOKY8g3/build/release/bin/kudu with pid 12633
2026-09-23T15:15:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2831 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5634 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275803 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST